Characterizing the Feasibility and Performance of Real-World Tumor Progression End Points and Their Association With Overall Survival in a Large Advanced Non-Small-Cell Lung Cancer Data Set.

PURPOSE: Large, generalizable real-world data can enhance traditional clinical trial results. The current study evaluates reliability, clinical relevance, and large-scale feasibility for a previously documented method with which to characterize cancer progression outcomes in advanced non-small-cell lung cancer from electronic health record (EHR) data.
METHODS: Patients who were diagnosed with advanced non-small-cell lung cancer between January 1, 2011, and February 28, 2018, with two or more EHR-documented visits and one or more systemic therapy line initiated were identified in Flatiron Health's longitudinal EHR-derived database. After institutional review board approval, we retrospectively characterized real-world progression (rwP) dates, with a random duplicate sample to ascertain interabstractor agreement. We calculated real-world progression-free survival, real-world time to progression, real-world time to next treatment, and overall survival (OS) using the Kaplan-Meier method (index date was the date of first-line therapy initiation), and correlations between OS and other end points were assessed at the patient level (Spearman's ρ).
RESULTS: Of 30,276 eligible patients,16,606 (55%) had one or more rwP event. Of these patients, 11,366 (68%) had subsequent death, treatment discontinuation, or new treatment initiation. Correlation of real-world progression-free survival with OS was moderate to high (Spearman's ρ, 0.76; 95% CI, 0.75 to 0.77; evaluable patients, n = 20,020), and for real-world time to progression correlation with OS was lower (Spearman's ρ, 0.69; 95% CI, 0.68 to 0.70; evaluable patients, n = 11,902). Interabstractor agreement on rwP occurrence was 0.94 (duplicate sample, n = 1,065) and on rwP date 0.85 (95% CI, 0.81 to 0.89; evaluable patients n = 358 [patients with two independent event captures within 30 days]). Median rwP abstraction time from individual EHRs was 18.0 minutes (interquartile range, 9.7 to 34.4 minutes).
CONCLUSION: We demonstrated that rwP-based end points correlate with OS, and that rwP curation from a large, contemporary EHR data set can be reliable, clinically relevant, and feasible on a large scale.

INTRODUCTION

Although overall survival (OS) is a benchmark of treatment efficacy in oncology, it may not be reliably determined during limited study periods. Intermediate tumor burden–based end points, such as progression-free survival (PFS), may evaluate clinically meaningful treatment effects more efficiently and have supported the regulatory approval of anticancer therapies.[1-3] Randomized controlled trials (RCTs) use the Response Evaluation Criteria in Solid Tumors (RECIST)[4] on imaging studies to determine tumor burden changes—for example, progression and/or response—which, in turn, affect clinical decisions and treatment benefit predictions.

Key Objective

The current study evaluated the large-scale feasibility of a previously described approach for the curation of a tumor progression variable from unstructured electronic health record (EHR)–based real-word data as well as the meaningfulness and reliability of the corresponding real-world progression-based end points.

Knowledge Generated

Our approach to real-world progression curation from EHRs is feasible and scalable to a database of more than 30,000 patients. Preliminary evaluations indicated that the associated progression end points are reliable and seem to be valid in the context of their relationship with clinical events documented in the EHR database from which they were generated and of published clinical trial results.

Relevance

We present a feasible and replicable approach to yield an EHR-generated progression variable that is clinically meaningful and that could be incorporated into large, contemporary real-world studies.

Whereas RCTs are a gold standard in clinical evidence, boundaries imposed by predefined eligibility criteria and sample size create a divide between the scope of an RCT and the disease population at large. Real-world data (RWD) that are gathered during routine care, such as in electronic health records (EHRs), can complement RCT results.[5] Collected retrospectively and prospectively from contemporary, generalizable, and often large cohorts as RWD, the resulting real-world evidence (RWE) can assess effectiveness, externally validating trial efficacy.[6]

Translating RCT end point standards into RWD may be challenging. In EHRs, tumor burden information may need to be abstracted from unstructured documents—for example, radiology reports and/or clinician notes—or inferred from structured data, such as laboratory findings and/or treatment orders. We previously found that applying RECIST retrospectively to EHRs—generally lacking raw images—is unfeasible as a result of incomplete and/or unclear real-world sources.[4] The question is how to unlock real-world clinical data as valid study end points.

To generate clinically meaningful conclusions, EHR-based characterization of relevant outcomes must accommodate RWD strengths and mitigate its limitations—for example, heterogeneous practice patterns and variable documentation. The approach needs to be transparent, replicable, and portable across data collection and clinical settings.

Our prior work compared multiple methods for curating real-world progression (rwP) from unstructured EHR documents.[7] Here, we evaluate one of those methods in a large, EHR-based cohort of patients with advanced non–small-cell lung cancer (aNSCLC), the leading cause of cancer mortality in the United States.[8] On the basis of a proposed development framework,[9] we examined three questions: Is capturing rwP retrospectively and with sufficient completeness feasible in large cohorts? Is rwP data sufficiently meaningful and reliable? and How do rwP-based end points—real-world PFS (rwPFS) and real-world time to progression (rwTTP)—compare with treatment-based end points, such as real-world time to next treatment (rwTTNT), including correlation with OS? Our objective was understanding the performance characteristics of rwP to enhance our ability to draw clinically meaningful RWD-based conclusions.

METHODS

For topics in this section, additional information can be found in the Appendix.

Data Sources

Flatiron Health’s longitudinal EHR-derived database contains aggregated, normalized, and harmonized deidentified patient-level data curated from structured and unstructured data via technology-enabled chart abstraction.[10] Normalization and harmonization involved mapping to a common terminology across different source systems. At the time of this analysis—April 2018—the database included more than 265 US cancer clinics using a variety of EHR systems and representing more than 2 million patients. Dates of death were sourced from a composite mortality variable comprised of EHR data linked to commercial mortality data and the Social Security Administration’s Death Master File.[11]

Institutional review board approval of the study protocol (#RWE-001; “The Flatiron Health Real-World Evidence Parent Protocol”; tracking: #FLI118044), with a waiver of informed consent, was obtained before study conduct, covering the data from all sites represented.

Cohort Selection

The cohort included patients who were diagnosed with aNSCLC—stage IIIB or metastatic stage IV, or recurrent advanced disease—between January 1, 2011, and February 28, 2018, with at least two EHR-documented clinical visits on or after January 1, 2011, and documentation of the initiation of at least one line of systemic therapy after advanced diagnosis (Fig 1). Patients who died 14 days or fewer from first-line systemic therapy initiation were excluded.

FIG 1.

Patients meeting all inclusion and exclusion criteria. The cohort was identified through a combination of structured data—for example, International Classification of Diseases (ICD) codes—and abstraction of unstructured documents. This resulted in 30,276 patients in the final cohort available for analysis. NSCLC, non–small-cell lung cancer.

rwP Definition

EHR data were curated to retrospectively determine rwP events using a previously proposed clinician-anchored abstraction approach (Appendix).[7]

The date of cancer progression was defined as the date of the first source evidence for progression referenced by the clinician—for example, radiology report, pathology, or exam—or the clinician note date when no evidence sources were documented.

The entire patient chart was abstracted, including multiple rwP events when present. This analysis focused on the first rwP event after systemic therapy initiation for the treatment of aNSCLC, with most thus occurring during first-line treatment. A continuous monthly audit to assess only interabstractor agreement was performed by duplicate abstracting a random subsample of 10% of newly accruing patients—that is, each new patient had an independent 10% chance of duplicate auditing—up to 100 patients per month.

Statistical Analyses

Calculation of real-world end points.

Real-world end points—rwPFS, rwTTP, rwTTNT, and OS—in patients with at least one relevant event (Table 1) were calculated using the Kaplan-Meier method,[12] indexed to the start of first-line systemic therapy. Kaplan-Meier curves and median time-to-event estimates were reported.

TABLE 1.

End Point Definitions and Censoring Approach

Feasibility.

We measured the completeness of data capture from imaging sources to assess the feasibility of abstracting rwP at scale from a large, generalizable EHR database, with a three-factor framework: Assessment, Documentation, and Abstraction (Fig 2).

FIG 2.

Framework for measuring the completeness of electronic health record (EHR) end point capture, such as the cancer status of progression. The ability to completely capture end points requires three dependent factors: Assessment, Documentation, and Abstraction. If any of the components of this framework—Assessment, Abstraction, or Evidence—are missing, capture of end points is incomplete. Our clinician-anchored real-world progression (rwP) curation approach focuses on the oncology clinician as the synthesizer of signals from the entire patient chart; imaging, when described in the oncology clinician note, is considered a piece of source evidence. (*) Other sources of missingness include abstractor error, missing documents in the EHR, etc.

The frequency of relevant imaging was estimated to determine whether patients were evaluated for cancer progression at expected intervals (Fig 2; Assessment). For an rwP event to be abstracted from the EHR, the clinician note referenced and documented disease status results (Fig 2; Documentation). Patients were censored at the time of death.

As a proxy for completeness of rwP capture by an abstractor (Fig 2; Abstraction), we measured inter-rater event agreement in the duplicate subgroup for the presence or absence of at least one progression event.

Abstraction time per patient chart was computed by the abstractor software interface. Average times were analyzed with descriptive statistics.

Meaningfulness and reliability.

Meaningfulness of the rwP data was evaluated using three criteria:

Face validity—the extent to which the method seems to measure the variable[13]: Descriptive survival analyses for rwPFS, rwTTP, and OS were determined and compared with similar statistics in the literature.

Association with near-term, clinically relevant downstream events, defined as death, start of a new therapy line, and stop or discontinuation of active systemic treatment: Near term was defined as 15 or fewer days before—allowing for short delays in documentation—and 60 or fewer days after—clinically appropriate for aNSCLC—the rwP date. Treatment discontinuation was defined as 60 days of active follow-up without structured documentation of additional systemic therapy.

Correlation of rwP-based end points—rwPFS and rwTTP—with OS, assessed at the patient level (Spearman’s ρ): For correlation calculations, the cohort was restricted to patients with a documented death event (n = 20,020 for rwPFS) or with both rwP and death events documented (n = 11,902 for rwTTP).

We assessed reliability by calculating agreement rates for the occurrence of an rwP event (event agreement), defined as the proportion of patients with an rwP event captured by two independent abstractors, and similarly for the documented event date (date agreement), in those with event agreement. Agreement for four date windows was also evaluated: 3, 7, 15, and 30 days.

Comparison with treatment-based end points.

rwP-based end points rwPFS and rwTTP were compared with rwTTNT, a treatment-based end point derived from structured data (Table 1). Correlation between rwTTNT and OS was calculated (Spearman’s ρ) among patients with both a documented death event and a second-line systemic treatment start (n = 9,269).

Sensitivity analyses.

We conducted sensitivity analyses to assess the robustness of the correlation between real-world end points—rwPFS, rwTTP, and rwTTNT—and OS. Three correlation calculations were reperformed and three additional sensitivity analyses were conducted (Appendix Table A1).

TABLE A1.

Sensitivity Analyses

RESULTS

Table 2 lists the demographic and clinical characteristics of the 30,276 eligible patients. Median follow-up after advanced diagnosis was 9.3 months (interquartile range, 4.3 to 19.4 months).

TABLE 2.

Demographic and Clinical Characteristics (N = 30,276)

Feasibility

Feasibility analyses were based on the completeness of the information collected in the eligible overall population (N = 30,276). The probability that a patient’s first, second, and third imaging set occurred within a 3-month window of first-line therapy start was 85%, 47%, and 18%, respectively. This increased to 94%, 89%, and 83%, respectively, for a 12-month window. At 3 months after aNSCLC diagnosis, 96% of patients had at least one clinician note. These frequencies of imaging and clinical assessments seemed to be consistent with realistic expectations, which suggested that there was no meaningful impact of missing evaluations on data completeness. Among duplicate abstracted patients (n = 1,065), agreement on the presence or absence of a first progression event was 94%, irrespective of event date.

Using the clinician-anchored approach, median time for an oncology-trained abstractor to abstract all EHR-documented rwP events for a patient was 18.0 minutes (interquartile range, 9.7 to 34.4 minutes).

Meaningfulness and Reliability

We conducted validity and reliability analyses in population subsets in which relevant events had been documented (Appendix Fig A1).

FIG A1.

Patient flow and evaluable samples within the study population. OS, overall survival; rwP, real-world progression; rwPFS, real-world progression-free survival; rwTTP, real-world time to progression.

Face validity.

We identified at least one rwP event after the initiation of systemic therapy for aNSCLC in 16,606 patients (55%). The proportion of patients with an rwP event increased among those who started second-line systemic treatment during the observation period (11,937 [87%] of 13,653) and among patients with at least 6 months of active follow-up from the start of the observation period (14 days after the initiation of first-line therapy; n = 16,025 [77%]). Figure 3 shows the Kaplan-Meier curves for rwPFS and rwTTP accompanied by OS.

FIG 3.

Kaplan-Meier survival curves for real-world progression-free survival (PFS), real-world time to progression (TTP), time to next treatment (TTNT), and overall survival (OS).

Association with near-term, clinically relevant, downstream events.

Among the evaluable population of 16,606 patients—those with at least one rwP event—11,366 patients (68%) had a documented near-term, downstream, clinically relevant event. In sensitivity analysis, which excluded follow-up losses within 60 days of an rwP event, the proportion with near-term, downstream events increased to 74%.

Correlation of rwP-based end points (rwPFS and rwTTP) with OS.

Table 3 lists the correlation between real-world end points and OS. Among patients who died during follow-up (n = 20,020), correlation between rwPFS and OS was moderately high (Spearman’s ρ, 0.76; 95% CI, 0.75 to 0.77). Among patients with both a progression event and death (n = 11,902), correlation between rwTTP and OS was lower (Spearman’s ρ, 0.69; 95% CI, 0.68 to 0.70).

TABLE 3.

Association Between OS and Real-World End Points (rwP and treatment based): Baseline and Sensitivity Analyses

Reliability.

As noted above, agreement on the occurrence of an rwP event in the duplicate abstraction patient subgroup (n = 1,065) without comparing any of the dates identified was 0.94 (Appendix Fig A2). Among those with at least one rwP captured by two independent abstractors (n = 358), agreement on the rwP date was 0.73 (95% CI, 0.68 to 0.78) when an exact date match was required, and 0.85 (95% CI, 0.81 to 0.89) within a 30-day date window. Reasons for date disagreement included ambiguous language (eg, “Progression on imaging last month”) or referencing different source evidence from different time points (eg, computed tomography chest v magnetic resonance imaging brain). Date agreement gains seemed to plateau at 7 to 15 days.

FIG A2.

Inter-rater date agreement among duplicate abstracted patients. Event agreement, which indicates whether abstractors agreed on the presence or absence of at least one progression event, was 0.94 (95% CI, 0.93 to 0.95). Date agreement for those patients with at least one real-world progression event captured by two independent abstractors (n = 358) is displayed as a function of the date window (in days) applied.

Comparison With Treatment-Based End Points

The rwTTNT Kaplan-Meier curve contour was visually similar to those of the progression-based end points (rwPFS and rwTTP; Fig 3). Median rwTTNT was 8.7 months (95% CI, 8.5 to 8.8 months), longer than that of median rwPFS (5.2 months; 95% CI, 5.1 to 5.3 months) and rwTTP (7.1 months; 95% CI, 7.0 to 7.3 months). Association between rwTTNT and OS was lower than that of rwPFS and rwTTP (Spearman’s ρ, 0.61; 95% CI, 0.60 to 0.62).

Sensitivity Analyses

For progression-based end points (rwPFS and rwTTP), sensitivity analyses demonstrated slight variability in OS correlations, but consistency with the primary analysis in magnitude and direction. The correlation between rwTTNT and OS was lower for some subgroups, dropping to 0.54 (95% CI, 0.52 to 0.56) when restricted to patients who started second-line therapy before 2015.

DISCUSSION

In oncology, tumor burden worsening—that is, progression—is a fundamental treatment metric. If tumor-based and time-to-event end points are to be characterized on the basis of real-world experience at the point of routine clinical care, a transparent, well-developed approach is needed to process the complexity—and volume—of EHR data. The current study extends a previous pilot study that demonstrated the feasibility of a clinician-anchored approach to abstract clinically relevant end points from EHR data.[7] We confirm reasonably high completeness, meaningfulness, and reliability of retrospectively EHR-abstracted rwP events for a cohort of more than 30,000 patients with aNSCLC.

Scalability addresses challenges unique to real-world studies, in particular, rapidly producing a large, reliable data set that is suitable for analysis. Our clinician-anchored rwP abstraction approach required less than 20 minutes per chart for most patients. Such factors as abstractor background and training and the patient’s clinical course—for example, number of scans—may affect abstraction time. With this approach, rwP was characterized for the original cohort of approximately 24,000 patients within 10 months (in the course of the study, the cohort subsequently grew to more than 30,000 patients). As long as imaging reports are available, this approach is EHR agnostic and therefore portable across data collection settings. Continued EHR and procedural advancements may further facilitate scaling and efficiency.

Of importance, there is no currently accepted framework for developing EHR-abstracted real-world end points for RWE studies, nor is there consensus on which criteria an end point would need to meet to clear a potential validation hurdle. We focused on three questions for our rwP variable: feasibility of the approach, reliability and meaningfulness, as well as comparison with other end points.

To address feasibility, we explored source data completeness. Evaluation metrics indicated that patients in our database seemed to be imaged and assessed by clinicians regularly, with an overall frequency that was consistent with our own practical experience. This finding suggests no meaningful impact of missing evaluations on data completeness.

We explored interabstractor consistency and correlation with downstream events, including mortality, as reliability and validity points. Consistency was high. Abstractors agreed on progression dates—within a reasonable window—85% of the time. Future studies may define reasonable windows or high agreement rates according to their specific clinical questions, disease settings, or desired level of certainty. Additional research may help understand the impact on estimates when agreement fluctuates across specific subgroups. Our results are clinically plausible in lung cancer. Approximately one third of patients did not receive a new treatment or experience another event after progression, and estimates of rwPFS, rwTTP, rwTTNT, and OS and Kaplan-Meier contours aligned with expectations for this population. Correlations between rwPFS and rwTTP and OS were similar or higher than clinical trial results.[14,15] The determination of whether the value of an rwP end point lies in estimating clinical benefit intrinsically or in correlating with traditional clinical trial benefit measures, such as PFS, will continue to unfold as technology enhancements expedite research in increasingly large and comprehensive data sets. Studies that replicate clinical trial findings using rwP and, where parallel, retrospective or prospective single data set collection of rwP and RECIST-based progression, which enables direct comparison, will be critical to determine when rwP is good enough.

For our third question, we compared our curated rwP-based end points with treatment-based end points, such as rwTTNT, that are readily obtainable from structured EHR information. rwTTNT captures treatment patterns, but unlike rwP it may conflate toxicity and progression as treatment change reasons, which may be valuable in some instances but suboptimal when the focus is strictly on tumor burden changes. Furthermore, rwTTNT calculation is negated for patients without subsequent treatment after progression. Of note, rwTTNT has a lower correlation with OS than rwPFS, which suggests that it is a weaker intermediate end point.

In the current study, rwP events were not associated with near-term clinically relevant downstream events for approximately one third of patients. A subset chart review confirmed no treatment changes despite evidence of worsening disease in a substantial portion of patients. In light of recent studies that have suggested a benefit from continuing immunotherapy postprogression, understanding potential differences between progression- and treatment-based end points is critical.[15-17] Ideally, a set of end points that measure multiple outcomes—for example, OS, PFS, and/or quality of life—will improve our understanding of patients’ cancer course.

Despite conceptual similarities, there are differences between rwP and RECIST-based progression, including the anchoring evidence source and assessment timing. Unlike the narrow RCT populations, the rwP variable performed within clinical expectations for a broad cohort of patients treated with a range of systemic therapies. Median rwPFS of approximately 5 months falls within the range observed in aNSCLC clinical trials for cytotoxic therapies, which the majority of this cohort received,[14,15,18] but is lower than that observed in recent immunotherapy and targeted therapy trials.[15,19-21] This trend may be more related to variations between populations—selected in clinical trials versus unselected in the real world—than between end points.

When data are not collected intentionally for research purposes, missing data, treatment compliance, measurement error, or unmeasured confounders create limitations and biases. In our case, additional work is needed to understand the potential impact of abstractor subjectivity or simplification in rwP assessment, as well as that of unbalanced evaluation frequency across groups. Because the study was restricted to patients with aNSCLC who were mostly from community settings, results may not be generalizable to other tumor types and patient populations. Our study did not include subgroup analyses that could require the abstraction of additional clinical variables from unstructured data (eg, extent of metastatic disease) and/or the incorporation of detailed structured data (eg, dose and timing of individual agents). As treatment options expand in aNSCLC, the association between rwPFS and OS may be affected by outcomes from subsequent therapy lines, by outlier exceptional responders to targeted therapy, or by the evolution of the progression concept in the immunotherapy era. Duplicate abstraction for agreement assessment ultimately represented a small portion of the final cohort size—approximately 1,000 patients of more than 30,000, of which only one third were evaluable for date agreement.

As survival rates have improved and more than one therapy line has become typical, OS has become an inadequate metric for ascertaining the benefit of some cancer treatment interventions. Intermediate end points are essential for evaluating treatment benefits in real-world contexts. This study presents a scalable, feasible, and replicable approach to yield an EHR-generated progression variable ready for incorporation into large, contemporary real-world analyses. More work is needed to document how rwP under different treatments relates to treatment effects observed in clinical trials; understand the impact of different imaging and assessment cadences—for example, informative censoring; and clarify the relationship between rwP and other end points, such as real-world tumor response. Similar work is also needed in other tumor types. Systematic assessment of real-world end points will improve confidence in RWD and resulting RWE.