Assessment of Alectinib vs Ceritinib in ALK-Positive Non-Small Cell Lung Cancer in Phase 2 Trials and in Real-world Data.

Importance: Quantitative assessment of bias from unmeasured confounding and missing data can help evaluate uncertainty in findings from indirect comparisons using real-world data (RWD). Objective: To compare the effectiveness of alectinib vs ceritinib in terms of overall survival (OS) in patients with ALK-positive, crizotinib-refractory, non-small cell lung cancer (NSCLC) and to assess the sensitivity of these findings to unmeasured confounding and missing data assumptions. Design, Setting, and Participants: This comparative effectiveness research study compared patients from 2 phase 2 alectinib trials and real-world patients. Patients were monitored from June 2013 to March 2020. Comparisons of interest were between alectinib trial data vs ceritinib RWD and alectinib RWD vs ceritinib RWD. RWD treatment groups were selected from nationally representative cancer data from US cancer clinics, the majority from community centers. Participants were ALK-positive patients aged 18 years or older with advanced NSCLC, prior exposure to crizotinib, and Eastern Cooperative Oncology Group Performance Status (PS) of 0 to 2. Data analysis was performed from October 2020 to March 2021. Exposures: Initiation of alectinib or ceritinib therapy. Main Outcomes and Measures: The main outcome was OS.
Results: In total, there were 355 patients: 183 (85 men [46.4%]) in the alectinib trial, 91 (43 men [47.3%]) in the ceritinib RWD group, and 81 (38 men [46.9%]) in the alectinib RWD group. Patients in the alectinib trial were younger (mean [SD] age, 52.53 [11.18] vs 57.97 [11.71] years), more heavily pretreated (mean [SD] number of prior therapy lines, 1.95 [0.72] vs 1.47 [0.81]), and had more favorable baseline ECOG PS (ECOG PS of 0 or 1, 165 patients [90.2%] vs 37 patients [77.1%]) than those in the ceritinib RWD group. The alectinib RWD group (mean [SD] age, 58.69 [11.26] years) had more patients with favorable ECOG PS (ECOG PS of 0 or 1, 49 patients [92.4%] vs 37 patients [77.1%]) and more White patients (56 patients [72.7%] vs 53 patients [62.4%]) compared with the ceritinib group. Compared with ceritinib RWD, alectinib-exposed patients had significantly longer OS in alectinib trials (adjusted hazard ratio [HR], 0.59; 95% CI, 0.44-0.75; P < .001) and alectinib RWD (HR, 0.46; 95% CI, 0.29-0.63; P < .001) after adjustment for baseline confounders. For the worst-case HR estimate of 0.59, residual confounding by a hypothetical confounder associated with mortality and treatment by a risk ratio greater than 2.24 was required to reverse the findings. Conclusions were robust to plausible deviations from random missingness for missing ECOG PS and underrecorded comorbidities and central nervous system metastases in RWD. Conclusions and Relevance: Alectinib exposure was associated with longer OS compared with ceritinib in patients with ALK-positive NSCLC, and only substantial levels of bias examined reversed the findings. These findings suggest that quantitative bias analysis can be a useful tool to address uncertainty of findings for decision-makers considering RWD.

Introduction

For rare indications, for which large sample sizes or randomized comparators are infeasible, real-world data (RWD) can at times be provided for consideration by decision-makers, such as health technology assessment (HTA) agencies, as an external comparator group for reimbursement decisions.[1,2] However, the risk of bias[3] in nonrandomized studies can pose a threat to their validity even when statistical methods to reduce confounding due to lack of treatment randomization are used, which has made HTA agencies reluctant to use them in their decision-making.[4,5] For example, RWD often contain significant amounts of missing observations for important prognostic variables that are not uniformly recorded in routine clinical practice, such as Eastern Cooperative Oncology Group (ECOG) Performance Status (PS) in cancer studies.[6] Findings from such studies can be prone to bias if assumptions about the mechanism of missingness are incorrectly specified or if unmeasured confounders are not accounted for.

One way to assess the sensitivity of adjusted measures of association of exposures and outcomes to systematic bias is to compute them over a range of assumptions about missing observations in RWD and unmeasured confounders. Then the robustness of real-world evidence can be evaluated according to how drastic or implausible these assumptions have to be for the observed association to be nullified. Quantitative bias analysis (QBA) methods can be used to identify the impact of missing data and residual confounding on results from nonrandomized studies using RWD.[7,8] In this comparative effectiveness study, we build on a previously reported indirect comparison of overall survival (OS) in patients with crizotinib-refractory, ALK-positive, non–small cell lung cancer (NSCLC) using data from single-group alectinib trials and ceritinib-exposed patients from RWD derived from a US database,[9] which was submitted to HTA agencies for reimbursement of alectinib.[10] In these assessments, concerns were raised about possible bias due to a large amount of missing ECOG PS in RWD and lack of sensitivity analyses to address the effect of unmeasured confounding.[11]

There is no ongoing or currently planned randomized trial comparing alectinib and ceritinib for the crizotinib-refractory, ALK-positive, NSCLC indication, and alectinib was approved for this indication in the US on the basis of results from 2 phase 2 clinical trials, NP28673 (ClinicalTrials.gov identifier NCT01801111)[12] and NP28671 (ClinicalTrials.gov identifier NCT01871805).[13] Thus, here we sought to (1) compare OS between patients from single-group alectinib trials and ceritinib-exposed patients in updated RWD (ceritinib RWD group) using propensity score weighting,[14] (2) compare OS between patients in RWD exposed to alectinib (alectinib RWD group) or ceritinib to measure real-world effectiveness of alectinib as an additional line of evidence to the trial data vs RWD comparison from the first objective, and (3) quantify the degree of residual confounding and assumptions about missing ECOG PS and underrecorded comorbidities and central nervous system (CNS) metastases that could nullify our findings from the first or second objectives using QBA.

Methods

This comparative effectiveness research study adheres to the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) reporting guideline and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies.[15] Approval for this study was granted by the Copernicus Group institutional review board. Informed consent was waived because the data were deidentified, in accordance with 45 CFR §46.

Study Population

At the time of initiation of alectinib or ceritinib (time 0 for start of follow-up), patients were aged 18 years or older, had a diagnosis of locally advanced or metastatic NSCLC, had a documented ALK rearrangement, and had previously received crizotinib. Patients were monitored from June 2013 to March 2020. RWD treatment groups were selected from nationally representative cancer data from 158 US cancer clinics, the majority from community centers. Patients with ECOG PS greater than 2 at baseline (measured <30 days before or <7 days after time 0) and American Joint Committee on Cancer stage below IIIB were excluded from all analyses. Race was self-reported by patients and was captured in the electronic health record. Race was assessed because it is a factor independently associated with lung cancer incidence and mortality, in part, as a result of differences in genetic susceptibility.[16] A flowchart of patient selection is shown in eFigure 1 in the Supplement. Additional detailed methods for this article can be found in the eAppendix in the Supplement.

Statistical Analysis

Along with hazard ratios (HRs), we plotted OS as a function of time using propensity score–weighted Kaplan-Meier curves, median survival times, and P values from log-rank tests. Two-sided P < .05 was considered statistically significant. All analyses were performed in R statistical software version 4.0.5 (R Project for Statistical Computing), and figures were generated in R and Python software version 3.6.9 (Python). Data analysis was performed from October 2020 to March 2021.

Results

Demographic and Clinical Characteristics

In total, there were 355 patients: 183 (85 men [46.4%]) in the alectinib trial, 91 (43 men [47.3%]) in the ceritinib RWD group, and 81 (38 men [46.9%]) in the alectinib RWD group. Compared with the ceritinib RWD group, patients in the alectinib trial group were younger (mean [SD] age, 57.97 [11.71] vs 52.53 [11.18] years), were more heavily pretreated (mean [SD] number of prior therapy lines, 1.47 [0.81] vs 1.95 [0.72] lines; prior chemotherapy, 39 patients [42.9%] vs 136 patients [74.3%]), and had more favorable ECOG PS (ECOG PS of 0 or 1, 37 patients [77.1%] vs 165 patients [90.2%]) (Table 1). A substantially larger number of patients in the alectinib trials than in the ceritinib RWD group had documented CNS metastases (112 patients [61.2%] vs 19 patients [20.9%]), which is possibly due, in part, to differences in recording between clinical trials and real-world clinical practice. ECOG PS was missing for 43 patients (47.3%) in the ceritinib RWD group; no patients in the trial had a missing baseline ECOG PS. Following reweighting, sufficient balance (standardized mean difference, <0.1) was achieved for all covariates (Table 2), and propensity weight distributions showed acceptable overlap with maximum values between weights of 1 and 2 (eFigure 2 in the Supplement). Fewer imbalances in baseline characteristics were identified between alectinib RWD and ceritinib RWD groups. Patients in the alectinib RWD group (mean [SD] age, 58.69 [11.26] years) were more likely to have favorable ECOG PS (ECOG PS of 0 or 1, 49 patients [92.4%] vs 37 patients [77.1%]), to be White (56 patients [72.7%] vs 53 patients [62.4%]), and to be smokers (36 patients [44.4%] vs 30 patients [32.9%]) compared with the ceritinib RWD group (Table 1). ECOG PS was missing for 28 patients (34.6%) in the alectinib RWD group. The only other variables with missing values in these 2 RWD groups were race (6 of 91 patients [6.6%] in the ceritinib group and 4 of 81 patients [4.9%] in the alectinib group) and cancer stage at diagnosis (1 of 91 patients [1.1%] in the ceritinib group and 0 of 81 patients in the alectinib group). Sufficient balance was achieved for all covariates after weighting and distributions of propensity weights almost completely overlapped (Table 2 and eFigure 2 in the Supplement).

Table 1.

Unadjusted Patient Characteristics by Treatment Group

Characteristic	Patients, No. (%)
	Ceritinib RWD (n = 91)	Alectinib trial data (n = 183)	SMDa	Alectinib RWD (n = 81)	SMDa
Follow-up time, median (IQR), mo	51.8 (38.8-61.8)	34.4 (31.9-38.0)	NA	29.9 (18.8-36.9)	NA
Age, mean (SD), y	57.97 (11.71)	52.53 (11.18)	0.475	58.69 (11.26)	0.063
Sex
Male	43 (47.3)	85 (46.4)	0.016	38 (46.9)	0.007
Female	48 (52.7)	98 (53.6)		43 (53.1)
White race	53 (62.4)	133 (73.9)	0.249	56 (72.7)	0.223
No smoking history	61 (67.0)	121 (66.1)	0.019	45 (55.6)	0.237
Histology, adenocarcinoma	87 (95.6)	175 (95.6)	0.001	78 (96.3)	0.035
Stage IV cancer	84 (93.3)	170 (92.9)	0.017	74 (91.4)	0.074
Eastern Cooperative Oncology Group Performance Status
0	18 (37.5)	64 (35.0)	0.405	21 (39.6)	0.452
1	19 (39.6)	101 (55.2)		28 (52.8)
2	11 (22.9)	18 (9.8)		4 (7.5)
Missing	43 (47.3)	0		28 (34.6)
Prior lines of therapy, mean (SD), No.	1.47 (0.81)	1.95 (0.72)	0.619	1.40 (0.72)	0.101
No prior chemotherapy	52 (57.1)	47 (25.7)	0.674	51 (63.0)	0.119
Central nervous system metastases	19 (20.9)	112 (61.2)	0.899	15 (18.5)	0.059

Abbreviations: NA, not applicable; RWD, real-world data; SMD, standardized mean difference.

SMDs comparing alectinib trial data or alectinib RWD with ceritinib RWD in the unadjusted data are shown. SMDs greater than 0.1 implied nonnegligible imbalance.

Table 2.

Propensity Score–Weighted Patient Characteristics by Treatment Group

Characteristic	Patients, weighted No. (%)
	Ceritinib RWD	Alectinib trial data	SMDa	Ceritinib RWD	Alectinib RWD	SMDa
Sample size (weighted)	273.4	266.5	NA	169.8	170.4	NA
Age, mean (SD), y	53.80 (11.49)	53.81 (11.47)	0.001	58.34 (11.97)	58.09 (11.43)	0.021
Male	126.2 (46.2)	126.4 (47.4)	0.025	79.4 (46.7)	79.9 (46.9)	0.004
White race	177.5 (64.9)	184.4 (69.2)	0.091	115.4 (67.9)	116.2 (68.2)	0.006
No smoking history	174.1 (63.7)	177.8 (66.7)	0.064	105.0 (61.9)	105.5 (61.9)	0.001
Histology, adenocarcinoma	256.6 (93.9)	255.3 (95.8)	0.085	162.9 (95.9)	163.9 (96.2)	0.014
Stage IV cancer	250.5 (91.6)	248.1 (93.1)	0.055	156.9 (92.4)	157.8 (92.6)	0.009
Eastern Cooperative Oncology Group Performance Status
0	98.6 (36.1)	100.3 (37.6)	0.032	66.9 (39.4)	66.1 (38.8)	0.016
1	145.5 (53.2)	138.3 (51.9)		81.6 (48.0)	82.2 (48.2)
2	29.3 (10.7)	27.9 (10.5)		21.3 (12.5)	22.1 (13.0)
Prior lines of therapy, mean (SD), No.	1.88 (1.09)	1.80 (0.73)	0.087	1.44 (0.77)	1.43 (0.77)	0.013
No prior chemotherapy	174.8 (63.9)	170.4 (63.9)	<0.001	67.8 (39.9)	67.3 (39.5)	0.009
Central nervous system metastases	NA	NA	NA	34.7 (20.4)	36.5 (21.4)	0.025

Abbreviations: NA, not applicable; RWD, real-world data; SMD, standardized mean difference.

SMD less than 0.1 implied negligible imbalance.

Treatment Effectiveness

Among patients with complete baseline confounder data (complete cases), alectinib was associated with significantly lower risk of all-cause mortality in both trial data vs RWD and RWD vs RWD comparisons. The adjusted HR for mortality comparing alectinib trial (180 patients) and ceritinib RWD (45 patients) was 0.55 (95% CI, 0.31-0.95; P = .03) (Table 3). Consistent with these results for trial data vs RWD comparisons, for the comparison of OS in alectinib RWD (52 patients) vs ceritinib RWD, the HR was 0.47 (95% CI, 0.23-0.96; P = .04) (Table 3). Unadjusted HRs were 0.64 (95% CI, 0.43-0.97) and 0.53 (95% CI, 0.30-0.92), respectively. Coefficients from the multivariable analyses are provided in eTable 1 and eTable 2 in the Supplement. The Little test for missing completely at random (MCAR) had a significant result, indicating that the MCAR assumption was generally invalid for the RWD.

Table 3.

Adjusted HRs From Complete-Case (Missing Completely at Random) and Multiple Imputation (Missing at Random) Analyses

Variable	Trial data vs RWD comparison				RWD vs RWD comparison
	Treatment group, alectinib trial	Comparator, ceritinib RWD	HR (95% CI)	P value	Treatment group, alectinib RWD	Comparator, ceritinib RWD	HR (95% CI)	P value
Complete case analysis
Sample size, No. of patients	180	45	0.55 (0.31-0.95)	.03	52	45	0.47 (0.23-0.96)	.04
Survival time, median (95% CI), mo	29.1 (21.2-39.6)	11.4 (8.5-NE)			43.3 (16.9-NE)	15.4 (8.8-37.5)
Multiple imputationa
Sample size, No. of patients	180	89	0.59 (0.44-0.75)	<.001	80	89	0.46 (0.29-0.63)	<.001
Survival time, median (95% CI), mo	29.1 (21.1-NE)	14.9 (9.1-35.0)			42.6 (24.4-NE)	17.84 (11.2-26.5)

Abbreviations: HR, hazard ratio; NE, not estimable; RWD, real-world data.

For multiple imputation, HRs were calculated by pooling.

Using multiple imputation to impute missing baseline confounder data, alectinib was similarly associated with a significantly lower risk of all-cause mortality in both trial data vs RWD and RWD vs RWD comparisons. The pooled adjusted HR comparing alectinib trial (180 patients) and ceritinib RWD (89 patients) was 0.59 (95% CI, 0.44 to 0.75; P < .001) (Table 3). In Kaplan-Meier curves, patients taking alectinib showed immediate improvement in OS after treatment initiation compared with those taking ceritinib and had significantly longer median survival times (29.1 months; 95% CI, 21.1 months to not estimable) compared with those taking ceritinib (14.9 months; 95% CI, 9.1 to 35.0 months). Sample adjusted Kaplan-Meier curves are shown in Figure 1.

Figure 1.

Adjusted Kaplan-Meier Curves for Overall Survival

Curves show survival for alectinib trial data vs ceritinib real-world data (RWD) (A) and alectinib RWD vs ceritinib RWD (B). Numbers at risk are reweighted sample sizes. Log-rank P values are shown.

Consistent with results from the alectinib trial group, patients in the alectinib RWD group had a significantly lower risk of all-cause death compared with patients in the ceritinib RWD group (HR, 0.46; 95% CI, 0.29 to 0.63; P < .001) (Table 3). Alectinib RWD were associated with significantly longer median survival time of 42.6 months (95% CI, 24.4 months to not estimable) compared with ceritinib RWD (17.8 months; 95% CI, 11.2 to 26.5 months). Adjustment for 11 additional baseline covariates, including comorbidities, all metastases, insurance status, and a separate Asian category for race (see the eAppendix in the Supplement for the full list), as a part of sensitivity analyses resulted in an HR of 0.60 (95% CI, 0.38-0.82). OS among alectinib trial and alectinib RWD was not significantly different (HR, 1.19; 95% CI, 0.83-1.56).

QBA of Unmeasured Confounding

To quantify the sensitivity of the adjusted worst-case HR from multiple imputation of 0.59 (95% CI, 0.44-0.75) to residual confounding, we computed the strength of associations of a hypothetical unmeasured binary confounder with mortality and treatment that would be required to nullify or reverse the observed beneficial effect of alectinib. We applied the method to the result with the point estimate closest to 1, which compared alectinib trial data with ceritinib RWD (HR, 0.59; 95% CI, 0.44-0.75). Results showed that the unmeasured confounder would need to be simultaneously highly correlated with mortality and imbalanced between treatment groups to reverse our findings (eFigure 3 in the Supplement). The E-value corresponding to HR of 0.59 (approximate risk ratio [RR], 0.69) was 2.24, representing confounder-mortality and confounder-treatment correlations needing to be simultaneously greater than 2.24 on the RR scale to move the HR point estimate to 1 and reverse our conclusions. This E-value was consistent with previous work using this data.[17] At the 95% CI limit of 0.75, the E-value was 1.74, representing the strength of residual confounding required to move the 95% CI to be intersecting with 1, and rendering results nonsignificant.

For comparison, the largest imbalance in data was observed for prior chemotherapy (RR, 1.73; mortality RR, approximately 1.10). ECOG PS was the most prognostic variable (mortality RR, approximately 2.1; association with treatment RR, 1.19). Smoking status, another important confounder in NSCLC,[18] had an association with treatment RR of 1.03 and mortality RR of 1.87.

QBA for Missing Data

The large amount of missing data for ECOG PS in the ceritinib group was a major source of concern for HTA appraisal of the previously submitted indirect comparison of alectinib trial and ceritinib RWD. To identify the distribution of missing ECOG PS in the ceritinib RWD that would be required to nullify or reverse the HR of 0.59 estimated missing at random (MAR) using multiple imputation, we performed a tipping point analysis by assuming that ECOG PS among ceritinib-treated patients in RWD could have been worse than expected under MAR (ie, missing not at random). To simulate worse-than-expected ECOG PS in the ceritinib RWD, we incorporated δ-shifts with multiple imputation.[19,20] As shown in eFigure 4 in the Supplement, negative values for δ-shifted ECOG PS in the ceritinib group to progressively poorer distributions, with progressively greater number of patients in the ceritinib group excluded as they were shifted to ECOG PS greater than 2 (an exclusion criterion for our study).

Tipping points for loss of statistical significance were not identifiable when comparing alectinib trial data with ceritinib RWD because of the inability to achieve balance beyond δ = −3 (Figure 2), which corresponded to a mean shift of 1.04 (first and third quartiles, 0.8 and 1.3) for imputed ECOG PS. At δ = −3, 0 individuals had an imputed ECOG PS of 0, which was 37% lower than observed among those with nonmissing ECOG PS and 40% lower than expected under MAR. Instead, we used the RWD vs RWD comparison, where balance was easier to achieve when changing ECOG distributions.

Figure 2.

Tipping Point Analysis for Missing Eastern Cooperative Oncology Group (ECOG) Performance Status (PS)

Tipping point analysis for varying δ shifts applied to missing baseline ECOG PS in the ceritinib RWD showing adjusted hazard ratios (HRs) comparing alectinib and ceritinib groups. Negative values of δ imply exponentially increasing odds of patients having poorer ECOG PS than expected under missing at random given their covariates. Sufficient balance for the trial data vs RWD comparison was not achieved beyond δ = −2.

For comparison of alectinib RWD with ceritinib RWD, the tipping point was identified at δ = −5 (Figure 2). A δ = −5 represents 148-fold odds of estimated ECOG PS in the ceritinib group being poorer than expected under MAR, or a mean shift of 1.63 to the estimated ECOG PS under MAR. For example, when δ = −5, a patient in the ceritinib RWD group with missing ECOG PS estimated to have a score of 0 conditional on their covariates, will instead be assigned a mean ECOG PS of 1.63. A δ = −5 corresponded to 0 patients with ECOG PS of 0, an approximately 20% decrease in number of patients with ECOG PS of 1 in the ceritinib RWD group and an additional 22 patients (22.5%) with ECOG PS shifted to greater than 2, who were excluded. HR point estimates plateaued between 0.75 and 0.85 with progressively worse ECOG PS shifts and did not reach 1 (Figure 2), indicating that our results could not be reversed under any scenario under missing not at random for ECOG PS, only lose their statistical significance.

In addition to ECOG PS, the distribution of comorbidities and metastases may have differed between trial data and RWD because of differences in surveillance and recording between trial and RWD. Sensitivity analyses found that our findings were robust to the assumption that, as a result of underreporting, CNS metastases and comorbidities actually had a 40% greater prevalence among those without a recorded CNS metastasis or comorbidity than reported in the ceritinib RWD.

Discussion

In settings where a randomized trial is infeasible or cannot be conducted in a timely manner, RWD can provide external control groups for head-to-head comparisons of treatment effectiveness.[21] Real-world evidence can also supplement results from clinical trials and support reimbursement decisions. Despite their utility, bias in nonrandomized studies can be a major concern and is one reason for their relative nonacceptance by HTA agencies. As a result, there is a need for sensitivity analyses of bias in nonrandomized studies to support their findings. This comparative effectiveness study is a follow-up on previous published work[9] comparing OS for alectinib trial vs ceritinib RWD in patients with crizotinib-refractory, ALK-positive NSCLC. We compared alectinib trial data and ceritinib RWD with updated data and included an additional comparison of ceritinib RWD with post–Food and Drug Administration approval alectinib RWD to support results from the trial data vs RWD comparison. Finally, we quantified the strength of residual confounding and missing data assumptions required to nullify our conclusions of a beneficial effect for alectinib for this indication.

Baseline patient characteristics between alectinib trial data and ceritinib RWD groups showed considerable imbalance in age and prior therapy. This is consistent with prior work using these data[9] and may be due to differences in patient selection in a clinical trial vs real-world practices captured in an electronic health record–derived database. There are likely additional unmeasured complex systematic differences in patient selection, variable capturing, coding, and recording (construct validity) that exist between trial data and RWD and that we have not addressed in this study. For example, the accuracy of ECOG PS is likely measured with differing accuracy between trial settings and routine care settings, leading to differential misclassification, and clinical equivalence of ECOG in the 2 data sources may be limited.[22] These scenarios could also be modeled using the methods presented within this study.

As an additional line of evidence, we compared alectinib RWD and ceritinib RWD groups, where patient data between treatment groups were likely better harmonized. As expected, we found fewer imbalances across baseline patient characteristics between the 2 RWD groups. ECOG PS was a key imbalance in both cases: ceritinib-treated patients in RWD had worse baseline ECOG PS than alectinib-treated patients in both the trial data and RWD. Although random chance may underlie this imbalance, it is also possible that patients with a more favorable performance status were preferentially channeled[23] to alectinib because of factors associated with clinical pathways and quality of care, such as perceived efficacy, toxicity, cost, ease of prescribing, and reimbursement.

After adjustment for baseline confounders, alectinib-treated patients had a lower risk of all-cause mortality than those treated with ceritinib. This statistically significant beneficial effect was consistent for both trial data vs RWD and RWD vs RWD comparisons using multiple imputation of missing baseline confounder data under MAR. HRs from complete-case analysis were also consistent with results from multiple imputation, suggesting that our findings were robust to assumptions of data MAR and MCAR. HRs in this study were similar to those in the study by Davies et al,[9] which estimated a mortality HR of 0.65 (95% CI, 0.48-0.88) comparing alectinib trial and ceritinib RWD, and were almost identical to their HR from imputation of 0.58 (95% CI, 0.44-0.76). Differences between our study and that of Davies et al[9] could be associated with the availability of additional patients and longer follow-up for some patients in the Flatiron Health database in our analysis, and our further harmonization of the definition of therapy lines between trial data and RWD.

For bias analysis, we first quantified the strength of residual confounding needed to reverse our findings. We found that for a hypothetical highly prognostic binary confounder to fully explain away a HR of 0.59 (95% CI, 0.44-0.75), it needed to be highly imbalanced between treatment groups. Our assumption of a binary confounder, as opposed to a continuous or polytomous one, was to simplify interpretability without loss of generality. On the basis of our estimates of correlation with mortality and imbalance between treatment groups for important measured prognostic variables such as ECOG PS and smoking history, we do not expect that sufficiently large systematic differences in unmeasured prognostic variables to reverse our findings were plausible.

We next explored the extent to which missing ECOG PS data would need to deviate from MAR to explain away our findings of a beneficial effect for alectinib. We focused on ECOG PS because it was highlighted by HTA agencies in their appraisal over concerns about the large amount of missing values and its high correlation with OS, but also because it may not be uniformly recorded across clinical practices and trials. We tested the assumption that patients with missing ECOG PS in the ceritinib RWD may have skewed toward poorer values than expected using multiple imputation under MAR, which may have resulted in confounding. To do this, we estimated HR and 95% CI over a range of distributions of ECOG PS in the ceritinib group by shifting progressively more patients to worse ECOG PS than would be estimated given other measured covariates. Because this shift in ECOG PS worsened the imbalance between alectinib trial data and ceritinib RWD and further reduced sample sizes as increasingly more patients were shifted to ECOG PS greater than 2 and, thus, were excluded, we were unable to identify tipping points when comparing alectinib trial data with ceritinib RWD because of the inability to achieve balance beyond δ = −3.

For RWD vs RWD comparisons, where balance was easier to achieve, the tipping point for loss of significance was δ = −5; no tipping point was identified for reversal of conclusions of a beneficial effect for alectinib because HR point estimates plateaued at approximately 0.8. The worst-case tipping point of δ = −3 corresponds to 20-fold greater odds of a poorer ECOG PS than expected under MAR in the ceritinib RWD group. More specifically, this corresponded to a mean ECOG PS shift of 1.04, and 0 individuals with ECOG PS of 0 among those missing baseline ECOG. This was 37% lower than observed among those with a nonmissing baseline ECOG PS, and 40% lower than expected under MAR. We expect that such an ECOG PS distribution in the ceritinib RWD by random chance is implausible. We also do not expect that large systematic differences in missingness mechanisms between treatment groups, which could have given rise to such data, should exist.

Strengths and Limitations

The strengths of this study include a nationally representative RWD sample and comparisons of both trial data and RWD for alectinib. As with any nonrandomized comparison, there is the possibility of unmeasured confounding. Although this study includes a rigorous assessment of bias due to unmeasured confounding and missing ECOG PS and underrecorded comorbidities and metastases, there may be additional multifactorial sources of bias that could have affected the reported estimates. However, given that comparisons of trial data vs RWD and RWD vs RWD produced consistent findings and that findings of a beneficial effect of alectinib from this study appear to be consistent with a retrospective analysis of progression-free survival and objective response in a cohort of patients in Taiwan for the same indication,[24] overall conclusions from this study are expected to be robust. Besides OS, there may be other patient-important measures of effectiveness (eg, toxicity) that were not examined in this study. However, prior retrospective analyses suggest that alectinib may be associated with fewer adverse events than ceritinib.[24] Note that unbiased estimation of treatment effects using weighting depends on correct specification of the propensity model. Although a doubly robust estimator was used to reduce dependence on this assumption, it is not guaranteed to provide unbiased estimates if both the propensity and outcome models were misspecified.

Conclusions

In conclusion, our results were consistent with prior studies and showed that alectinib was associated with longer OS compared with ceritinib in patients with crizotinib-refractory, ALK-positive NSCLC. Application of QBA methods showed that our findings were robust to plausible sources of bias from unmeasured confounding and distributional assumptions about missing baseline ECOG PS and underrecording of comorbidities. There may be additional sources of bias that we have not addressed in this study, such as from misclassification of OS, which are especially relevant for comparison of single-group trial data and RWD. We propose that transparent and rigorous quantitative evaluation of uncertainty under a range of scenarios and sources of bias, and a discussion of their plausibility, should accompany submissions of nonrandomized studies to aid regulatory and reimbursement decision-making.