Concordance of Clinician-Documented and Imaging Response in Patients With Stage IV Non-Small Cell Lung Cancer Treated With First-Line Therapy.

Importance: In observational oncology studies of solid tumors, response to treatment can be evaluated based on electronic health record (EHR) documentation (clinician-assessed response [CAR]), an approach different from standardized radiologist-measured response (Response Evaluation Criteria in Solid Tumours [RECIST] 1.1). Objective: To evaluate the feasibility of an imaging response based on RECIST (IRb-RECIST) and the concordance between CAR and imaging response based on RECIST assessments, and investigate discordance causes. Design, Setting, and Participants: This cohort study used an EHR-derived, deidentified database that included patients with stage IV non-small cell lung cancer (NSCLC) diagnosed between January 1, 2011, to June 30, 2019, selected from 3 study sites. Data analysis was conducted in August, 2020. Exposures: Undergoing first-line therapy and imaging assessments of response to treatment. Main Outcomes and Measures: In this study, CAR assessments (referred to in prior publications as "real-world response" [rwR]) were defined as clinician-documented changes in disease burden at radiologic evaluation time points; they were abstracted manually and assigned to response categories. The RECIST-based assessments accommodated routine practice patterns by using a modified version of RECIST 1.1 (IRb-RECIST), with independent radiology reads. Concordance was calculated as the percent agreement across all response categories and across a dichotomous stratification (response [complete or partial] vs no response), unconfirmed or confirmed.
Results: This study found that, in 100 patients evaluated for concordance, agreement between CAR and IRb-RECIST was 71% (95% CI, 61%-80%), and 74% (95% CI, 64%-82%) for confirmed and unconfirmed response, respectively. There were more responders using CAR than IRb-RECIST (40 vs 29 with confirmation; 64 vs 43 without confirmation). The main sources of discordance were the different use of thresholds for tumor size changes by RECIST vs routine care, and unavailable baseline or follow-up scans resulting in inconsistent anatomic coverage over time. Conclusions and Relevance: In this cohort study of patients with stage IV NSCLC, we collected routine-care imaging, showing the feasibility of response evaluation using IRb-RECIST criteria with independent centralized review. Concordance between CAR and centralized IRb-RECIST was moderate. Future work is needed to evaluate the generalizability of these results to broader populations, and investigate concordance in other clinical settings.

Introduction

Data collection on patient response to treatment outside of the clinical trial setting and the evidence it generates are valuable areas of research to complement and supplement clinical trials.[1] Such evidence can address issues such as generalizability to diverse populations, or investigation of rare populations, becoming a component of clinical development programs and/or regulatory decision making.[2,3]

These types of data analyses require metrics and end points distinct from clinical trials to accommodate particularities such as source heterogeneity or missing information.[4] In clinical trials of solid tumors in oncology, treatment response or disease progression end points are typically measured via RECIST as standard research practice.[5] In contrast, the definition and validation of response or progression end points for observational studies is an area of active investigation.[4,6,7,8]

Our work focused on deidentified EHR-derived data; previously, our team characterized a response variable based on clinician-assessed response (CAR, referred to in prior publications as “real-world response”).[9] This variable, derived from EHR notes anchored at imaging assessment time points throughout lines of systemic therapy, relies on the treating clinician’s written synopses of patients’ status, not solely on radiology reports. We compared cohort-level response-rate estimates using clinical trial RECIST and CAR on analogous patient cohorts side by side,[9] and investigated the correlation between end point results with either variable, accounting for differences in cohort-level patient characteristics.[9] The CAR variable could provide clinically meaningful information comparable to RECIST at the cohort level, but the relationship between CAR and standardized, centralized, and radiologist-measured response (RECIST 1.1) at the patient-level within the same cohort remained an open question.

We report the results from Retrospective Unscheduled scan collection assessing disease Burden, a non-Interventional Endpoint Snapshot (RUBIES), a study aimed to further characterize the relationship between a RECIST-based response with independent review and CAR in patients with stage IV non–small cell lung cancer (mNSCLC). Our objectives were (1) to develop and examine the feasibility of conducting centralized imaging response based on RECIST (IRb-RECIST) in community settings, (2) to evaluate the patient-level concordance between CAR and IRb-RECIST evaluations performed on routine-care imaging, as a first step to assess the degree of comparability of both variables, and (3) to investigate causes of discordance, to better understand the nature of the clinical information captured by the CAR variable. We focused our work on a patient cohort engineered to optimize the availability of documentation necessary for our analyses.

Methods

Data Source

This retrospective observational study used the Flatiron Health nationwide longitudinal deidentified EHR-derived database,[10] comprising patient-level structured and unstructured data, curated via technology-enabled abstraction.[11] During the study period, overall data in the database originated from approximately 280 US cancer clinics (approximately 800 sites of care). Institutional review board (IRB) approval of the study protocol, with a waiver of informed consent, was obtained prior to study conduct, covering the data from all sites represented. Approval was granted by the WCG IRB. (The Flatiron Health Real-World Evidence Parent Protocol, Tracking # FLI118044). The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

Cohort Selection

The first step of cohort selection was to apply a series of feasibility criteria. Patients whose EHRs originated from 3 study sites (community oncology practices) were included if they had a stage IV diagnosis between January 1, 2011, and June 30, 2019, had documentation of treatment (first-line therapy, 1L), and at least 2 EHR-documented clinical visits on or after January 1, 2011. Patients without structured activity (ie, a visit, vital, medication order, or medication administration) within 90 days after the initial metastatic diagnosis date were excluded (Figure).

Figure.

Patient Sample Flow Through Cohort Selection Steps

CAR Indicates clinician-assessed response; EHR, electronic health record; NSCLC, non–small cell lung cancer.

aAt least 1 baseline CT covering any of the chest, abdomen, or pelvis (CAP), and at least 1 postbaseline scan of any modality of interest.

bAt least 1 baseline CT covering any of the chest, abdomen, or pelvis (CAP), and at least 1 postbaseline scan of any modality of interest.

cAttrition in some instances was owing to operational constraints, when study staff may not have been able to go through all patients with EHR documentation of scans.

Within the cohort generated via feasibility criteria, we applied additional criteria as a second step, the documented-scan criteria. These included having EHR documentation of at least 1 chest, abdomen, or pelvis (C/A/P) computed tomography (CT) scan performed within 6 weeks prior to 1L treatment initiation (ie, baseline window). Patients were also required to have at least 1 EHR-documented scan performed any time during 1L treatment (ie, follow-up window). To define 1L treatment, receipt of therapy was confirmed by manual abstraction. Qualifying images included CT, bone scans, positron emission tomography (PET)-CT, and magnetic resonance imaging (MRI), regardless of contrast dye administration.

In a subsequent step, a third set of criteria (not included in the prior steps) required patients to have at least 1 C/A/P CT scan available in the baseline window and at least 1 scan of any qualifying modality as described above in the follow-up window. All available digital images from patients with documented scans (as per the documented-scan criteria) were obtained retrospectively from the treating centers directly and/or via picture archiving and communication systems (PACS), and were uploaded into a viewing system in the Digital Imaging and Communications in Medicine (DICOM) format. Deidentified study IDs were entered for each patient to ensure attribution and traceability. These were the available-scan criteria.

To be eligible for the concordance analysis, patients had to meet both the documented-scan and the available-scan criteria, and had to have at least 1 clinician assessment of tumor burden documented in the EHR following radiographic evaluations (CAR assessment).

Variables and Assessments

Imaging Response Based on RECIST (IRb-RECIST)

A radiology charter was developed by the study team, implementing modifications to apply RECIST 1.1[5] to radiologic images collected retrospectively during routine care, aiming to balance feasibility improvements with minimal effect on RECIST performance, based on clinical judgment (Table 1).

Table 1.

Key Features in the Application of RECIST 1.1 to Retrospectively Collected Radiologic Images Performed as Part of Routine Care

RECIST 1.1	Imaging response based on RECIST	Observed deviations from RECIST 1.1 and potential effect
Lesions are classified as measurable/nonmeasurable.	Same.	NA.
Baseline method of assessment must be followed throughout the study/observation period. Lesions on chest x-ray may be considered measurable if they are clearly defined and surrounded by aerated lung.	Baseline method of assessment might not be followed for the duration of the therapy of interest.a Chest x-rays are largely unavailable in PACS or not requested. Lesions identified on chest x-ray will be selected as nontarget lesions only.	Inconsistent anatomic coverage across baseline and follow-up assessments. Might result in false determinations of response or not evaluable time points (discordant cases were 26 of 84 and 3 of 16 in the patients with and without any inconsistent anatomic coverage respectively). Inconsistent imaging modalities. May result in suboptimal readouts (discordant cases were 12 of 32 and 17 of 68 in patients with and without inconsistent imaging respectively).
An assessment of tumor burden at baseline must be performed; target and nontarget lesions must be selected and documented. (In clinical trials in patients with NSCLC, the typical specified baseline period is within 28 days of treatment initiation).	A baseline assessment is required, target and nontarget lesions are selected and documented, but baseline window is defined as 6 weeks to allow a more comprehensive capture of scans and a larger cohort.	Could not quantify.
Response categories (complete, partial, stable disease, progressive disease) classified based on measurement thresholds for target lesions, general size assessment for nontarget lesions, and the emergence of new lesions.	Same.	NA.
Assessment time points prespecified in protocol. Outside these prespecified points, assessments are computed as unscheduled assessments.	Timing of follow-up assessments is not prespecified in protocol. No concept of unscheduled assessments was applied.	Could not quantify in our retrospective cohort, where frequency was at the discretion of the treating clinicians.
Confirmation of response (for PR, CR) mandated when response rate is a main trial endpoint, with a dedicated assessment at least 4 weeks after observing initial response.	Confirmation of response requires the availability of a subsequent documentation of response no earlier than 4 weeks from the time a CR or PR is first suspected.	Missing subsequent response confirmation time points after initial response, n = 3 (3%). Might result in misclassification as nonresponders.

Abbreviations: CR, complete response; NA, not applicable; NSCLC, non–small cell lung cancer; PR, partial response.

Qualifying images included computed tomographic, bone scans, positron emission tomography, and magnetic resonance imaging, regardless of contrast dye administration.

Baseline method of assessment might not be followed for the duration of the therapy of interest.

Scans within 14 days of an initial scan during follow-up were bundled and read as a single assessment time point at the earliest date because unique assessments may routinely span a succession of multiple scans (eFigure in the Supplement). The independent centralized review of available images, per the radiology charter, used 2 primary readers plus adjudication. Response was categorized based on lesion measurement and presence or absence of new lesions at each follow-up assessment. The best overall response was the best response recorded from the start of treatment until the earliest of progression or the end of treatment, taking into account any requirement for confirmation, described herein. At each time point, digital images were assessed for readability (as readable, readable but suboptimal, not readable) according to radiology standards established by the review committee.

Clinician-Assessed Response

Clinician-assessed response (referred to in prior publications as “real-world response” [rwR])[9] was defined as a change in disease burden documented in the EHR by the treating clinician at time points associated with radiologic evaluations during a line of therapy, abstracted manually from EHR documentation, and classified into real-world response categories: complete response, partial response, stable disease, progressive disease, indeterminate response, and not documented.[9] Response classifications could refer to preceding assessments for comparison (not necessarily to baseline, attempting to reflect routine care practices).[9] Response confirmation was based on documentation of continued response in subsequent time points (which could be noted as “stable” referring to an immediately prior response [partial or complete] documented in the EHR).

Best overall response was the best time point response recorded from the treatment start until the earliest date of progression or the end of treatment, taking into account a requirement for confirmation when applicable.[9]

Statistical Analyses

Scan Availability and Readability Analyses

We assessed the feasibility of the IRb-RECIST approach by calculating the rate of patient-level scan availability in the cohort meeting the feasibility criteria. Feasibility for a patient was defined as (1) having 1L therapy confirmed by abstraction, and (2) meeting both documented-scan and available-scan criteria.

For the cohort where the IRb-RECIST approach was feasible and CAR assessments were also available (ie, concordance cohort), we estimated imaging-level scan availability and rate of readability. Imaging-level scan availability was defined as the percentage of EHR-documented scans that had a corresponding digital image available for radiology reads. Readability rate was based on the proportion of available scans that yielded a time point response readout by the centralized readers.

Concordance Analysis

Patients meeting the documented and available scan criteria, and with at least 1 CAR assessment, were included in the concordance analysis. Analyses were reported at the patient level, where we calculated the agreement between paired assessments of CAR and IRb-RECIST on the same patient using percent agreement and Cohen’s κ (with point estimates and 95% CIs). As the primary outcome of the study, the calculations used a dichotomous response outcome (response [CR/PR] vs no response [non-CR/PR]); we also performed the analysis using an all categories best response outcome, and using a dichotomous disease control outcome (ever achieved SD or better vs not achieving at least SD). For all patient-level response outcomes, we performed analyses on the basis of unconfirmed and confirmed responses. Sensitivity analyses were performed on the dichotomous and best response outcome stratified by therapy class and availability of scans.

An exploratory qualitative review of EHR charts was performed for the cases with disagreement between CAR and IRb-RECIST assessments, to catalog any sources of discordance.

Analyses were conducted in R statistical software (version 3.6.1, R Foundation). The analysis was conducted in August, 2020.

Results

Feasibility criteria were met by 1210 patients; 100 patients were included in the concordance analysis (Figure). Study cohorts were largely similar to the broader advanced NSCLC population in the originating EHR-derived longitudinal database, with a median age at diagnosis between 67 to 69 years, and predominance of patients with tumors of nonsquamous histology and of smokers (Table 2). There were greater proportions of White patients and patients with nonsquamous cell carcinomas in the concordance cohort (Table 2; eTable 1 in the Supplement).

Table 2.

Characteristics of the Patient Cohort for the Concordance Study

Characteristics	No. (%)
	Feasibility cohort (n = 1210)	Concordance cohort (n = 100)
Age at stage IV diagnosis, median (IQR), y	69.0 (62.0-76.0)	67.5 (60.8-76.0)
Sex
Female	591 (48.8)	51 (51.0)
Male	619 (51.2)	49 (49.0)
Race
Asian	31 (2.6)	3 (3.0)
Black/African American	42 (3.5)	1 (1.0)
Unknowna	155 (12.8)	6 (6.0)
White	886 (73.2)	84 (84.0)
Otherb	96 (7.9)	6 (6.0)
Histologic findings
Nonsquamous cell	919 (76.0)	83 (83.0)
NOS	49 (4.0)	0
Squamous cell	242 (20.0)	17 (17.0)
Smoking history
Yes	992 (82.0)	82 (82.0)
No	215 (17.8)	18 (18.0)
Unknown	3 (0.2)	0
Stage IV diagnosis year
2011-2014	497 (41.1)	36 (36.0)
2015-2017	501 (41.4)	49 (49.0)
≥2018	212 (17.5)	15 (15.0)
EGFR variant
Tested	585 (48.3)	54 (54.0)
Variation negative	433 (74.0)	37 (68.5)
Variation positive	114 (19.5)	13 (24.1)
Other	38 (6.5)	4 (7.4)
ALK alteration
Tested	580 (47.9)	52 (52.0)
Rearrangement not present	496 (85.5)	48 (92.3)
Rearrangement present	25 (4.3)	3 (5.8)
Other	59 (10.2)	1 (1.9)
KRAS variant
Tested	195 (16.1)	17 (17.0)
Variation negative	121 (62.1)	7 (41.2)
Variation positive	62 (31.8)	9 (52.9)
Other	12 (6.2)	1 (5.9)
1L therapy
ALK inhibitor	22 (1.8)	3 (3.0)
Anti–VEGF-containing	192 (15.9)	27 (27.0)
Clinical study	6 (0.5)	0
EGFR TKI	128 (10.6)	19 (19.0)
Anti-EGFR antibody-based	3 (0.2)	1 (1.0)
Nonplatinum-based chemoth comb	3 (0.2)	0
Other	1 (0.1)	0
PD-(L)1-based	182 (15.1)	15 (15.0)
Platinum-based chemoth comb	631 (52.4)	35 (35.0)
Single-agent chemotherapy	37 (3.1)	0

Abbreviations: ALK, anaplastic lymphoma kinase; EGFR, epidermal growth factor receptor; IQR, interquartile range; NOS, not otherwise specified; PD-(L)1, programmed cell death-(ligand) 1; TKI, tyrosine kinase inhibitor; VEGF, vascular endothelial growth factor.

Patients whose race was either not captured or who only had “Hispanic or Latino” ethnicity captured without race, were categorized as “Unknown” owing to small cohort size.

Patients with race category not listed above were categorized as "Other."

Scan Availability and Readability Analyses

Of 1210 patients meeting the feasibility criteria, we selected an upfront sample of 672 for manual abstraction to efficiently confirm receipt of therapy (1L); confirmation was possible for 652 (97%) patients. Of them, 302 (46%) patients met the documented scan criteria, and 75 (12%) met both the documented and available scan criteria. Of the 538 patients not sampled for upfront abstraction, 29 met both the documented and available scan criteria and their receipt of therapy was subsequently confirmed by manual abstraction. Of these 104 patients who met both the documented and available scan criteria, 100 patients had at least 1 CAR assessment in the EHR.

There were a total of 1259 EHR-documented scans, of which 1002 (80%) were available for centralized IRb-RECIST reads; 322 of 434 (74%) of the baseline scans and 680 of 825 (82%) of the follow-up scans were available. Imaging was most frequently unavailable when it had been performed in a radiology facility separate from the oncology clinic. In addition, 177 scans were available for IRb-RECIST reads but not documented in the EHR.

As described in the methods, scans were bundled into assessment time points (within 14-day windows) in the concordance cohort. Few time points (3 [<1%]) had scans classified as unreadable, 332 (34%) as suboptimal but readable, and 629 (65%) as optimal by the independent radiology reading facility.

Concordance Results

In the cohort of patients eligible for the concordance analysis (N = 100), percent agreement between CAR and IRb-RECIST based on the dichotomous analysis (response/no-response) categorization with confirmation was 71% (95% CI, 61%-80%); unconfirmed response results were similar, 74% (95% CI, 64%, 82%). Percent agreement was lower in the analysis based on best response categories, 52% (95% CI, 42%-62%) (Table 3; eTable 2 in the Supplement).

Table 3.

Concordance Results From the Analyses Across All Categories, Best Response, and in the Dichotomized Response/Nonresponse Categories

Variable		CAR [n = 100], No (%)								All categories	Dichotomized
		CR	PR	(Dichot) response	SD	PD	Indeter	Not doc	(Dichot) no-response
Imaging response based on RECIST (n = 100)	CR	0 (0.0)	2 (2.0)	NA	1 (1.0)	0	0	0	NA	Agreement, % (95% CI): 52 (41.8-62.1)	Agreement, % (95% CI): 71a (61.1-79.6)
	PR	1 (1.0)	17 (17.0)		8 (8.0)	0	0	0		κ statistics (95% CI): 0.29 (0.14-0.43)	κ statistics (95% CI): 0.37b (0.17-0.56)
	(Dichot) response	NA	NA	20 (20.0)	NA	NA	NA	NA	9 (9.0)	NA	NA
	SD	1 (1.0)	15 (15.0)	NA	24 (24.0)	8 (8.0)	0	1 (1.0)	NA	Unconfirmed agreement, % (95% CI): 58 (47.7-67.8)	Unconfirmed agreement, % (95% CI): 74 (64.3-82.3)
	PD	0	4 (4.0)		5 (5.0)	11 (11.0)	0	0
	NE	0	0		0	2 (2.0)	0	0		κ statistics (95% CI): 0.37 (0.22-0.52)	κ statistics (95% CI): 0.50 (0.33-0.67)
	(Dichot) no-response	NA	NA	20 (20.0)	NA	NA	NA	NA	51 (51.0)	NA	NA

Abbreviations: CAR, clinician-assessed response; CR, complete response; NA, not applicable; NE, not evaluable; PD, progressive disease; PR, partial response; SD, stable disease.

In the analysis based on dichotomous disease control outcome (ever achieved SD or better vs not achieving at least SD), agreement was 82% (95% CI, 73.1%-89.0%) and κ statistics, 0.48 (95% CI, 0.25-0.70).

Analyses based on confirmed response, except when noted (N = 100).

The qualitative EHR medical chart review revealed 2 potential drivers of discordance between CAR and IRb-RECIST assessments in the dichotomous analysis: lack of standardized quantitative thresholds in the assessments by treating clinicians, which resulted in different meanings for partial response as a CAR category compared with the IRb-RECIST category, and unavailability of baseline or follow-up scans (eTable 3 in the Supplement; see Table 1 for the potential effect of some of the RECIST criteria adaptations on the subsequent concordance analysis).

Sensitivity/Subgroup Analyses

Analyses showed that concordance between CAR and IRb-RECIST varied across different therapy classes (Table 4), highest for chemotherapy and lowest for targeted and immunotherapy. This pattern was consistent in the dichotomous and best-response stratifications, with response confirmation.

Table 4.

Sensitivity Analyses of Concordance Rates (by Percent Agreement) According to Treatment/Therapeutic Class and to Scan Availability

Analysis	No.	% (95% CI)
		Best response categories		Dichotomized response, y/n
		Unconfirmed	Confirmeda	Unconfirmed	Confirmeda
According to therapy class
Anti–VEGF-containing	27	63 (42.4-80.6)	51.9 (31.9-71.3)	81.5 (61.9-93.7)	70.4 (49.8-86.2)
Chemotherapy	35	54.3 (36.6-71.2)	62.9 (44.9-78.5)	71.4 (53.7-85.4)	82.9 (66.4-93.4)
Targeted	23	65.2 (42.7-83.6)	43.5 (23.2-65.5)	78.3 (56.3-92.5)	56.5 (34.5-76.8)
PD-1/PD-L1-based	15	46.7 (21.3-73.4)	40 (16.3-67.7)	60 (32.3-83.7)	66.7 (38.4-88.2)
According to scan availability
All EHR-documented scans available	23	60.9 (38.5-80.3)	60.9 (38.5-80.3)	73.9 (51.6-89.8)	73.9 (51.6-89.8)
At least 1 EHR-documented scan unavailable	77	57.1 (45.4-68.4)	49.4 (37.8-61)	74.0 (62.8-83.4)	70.1 (58.6-80)

Abbreviations: EHR, electronic health record; PD-(L)1, programmed cell death-(ligand) 1; Targeted therapy, anaplastic lymphoma kinase (ALK) and epidermal growth factor receptor (EGFR) tyrosine kinase inhibitors (TKIs), and anti-EGFR antibody-based; VEGF, vascular endothelial growth factor.

Patients with an initial response, but without subsequent scan to confirm it, would be considered unconfirmed.

A different subanalysis investigated agreement rates according to availability of scans. Concordance was greater among patients with full availability vs patients with at least 1 unavailable scan result (Table 4; eTable 2 in the Supplement).

Discussion

This study is one of the first to systematically collect routine care solid tumor imaging from EHR-derived data sources to show the feasibility of implementing a RECIST adaptation using independent centralized imaging review (IRb-RECIST), which was then used to benchmark a response variable derived from clinician-documented response in the EHR. These 2 approaches had moderate concordance overall in the patient cohort studied, possibly affected by factors such as treatment type or availability of scans.

The CAR evaluation yielded more responders compared with centralized IRb-RECIST assessment. This may be owing to reasons intrinsic to the CAR variable, such as treating clinicians not relying on stringent response classification thresholds relative to a baseline, but rather on holistic clinical judgments. These features probably compound more general discrepancy patterns between local-investigator and centralized assessments, well documented in oncology studies.[12,13,14,15,16,17]

Overall, a formal requirement for response confirmation did not affect concordance. Given that dedicated confirmation assessments are not typical in routine practice (where surveillance aims to monitor response durability rather than confirmation), this lack of effect mitigates a potential concern that could have invalidated the CAR variable. The variability according to therapy class may be owing to the different benefit depth and duration across therapies, as well as the different routine assessment frequencies for different therapies.

Other imaging-based studies are actively developing methodology for oncology RWD analyses. Kehl et al[8] focused specifically on radiology reports to develop a machine-learning model capable of annotating imaging reports to categorize response or progression. They did not examine their model’s performance relative to RECIST, and therefore cannot be compared with our report; however, their work is an important step to address the importance of scalability in the analysis of large RWD samples.

Feinberg et al[18] applied RECIST to clinical imaging in EHR-based sources and to associated clinicians’ assessments. They used an adaptation of RECIST 1.1 for real-world application, but their study and ours differ across several points: Feinberg et al opted for local rwRECIST assessment by treating clinicians rather than centralized review, they allowed rwRECIST application on available imaging reports in addition to images themselves, and they evaluated single, best response follow-up time points for each patient (as opposed to all assessment time points). Interestingly, some of their findings, such as clinicians’ assessments yielding more responders relative to their rwRECIST adaptation, are consistent with ours, probably owing to similar factors.

Limitations

Among the study limitations are, first, its scope, as defined by cohort size, number and type of study sites (3 community clinics), and clinical setting (single tumor type, single therapy line). We provide context about the representativeness of the study cohort, but these findings on imaging availability may be highly specific to the 3 study sites and to the study processes; indeed, we cannot rule out selection bias associated to the imaging availability requirements with an effect on concordance, although our sensitivity analyses suggest that the association was probably minor. Investigating the generalizability of these results by expanding the scope of future research is therefore warranted. Second, scan availability and consistency were variable (in terms of imaging modalities, contrast administration, anatomic coverage and timing) despite our eligibility criteria, illustrating challenges associated with data collection from routine clinical care. We used an IRb-RECIST approach (more flexible than RECIST 1.1) as a benchmark; therefore, our reference was not strictly equal to RECIST clinical trial methodology, and the precise effect of our adaptations is not known. Applying criteria and standards conceived for clinical trials to RWD may pose a challenge. Imaging-based studies, like this, may require calibrations of stringency on technical or timing requirements, to maintain inclusive cohorts and maximize data availability, followed up by deeper research to characterize the analytic effect of those calibrations. Understanding which factors affect concordance will guide data collection improvements and iterations that could mitigate that effect. Nonetheless, in this study we were able to conduct centralized imaging readouts by modifying RECIST; sensitivity analyses that assessed the effect of data heterogeneity produced largely consistent results with the main analyses, indicating that the confounding effects associated with missing or inconsistent imaging were probably not major.

The implications of this study are 2-fold. In terms of feasibility, we believe our modifications to the RECIST approach were largely successful to enable centralized real-world imaging review. At the same time, we acknowledge that a relatively large initial patient sample was required to arrive at the concordance-analysis sample. Adaptation of clinical trial standards such as this may provide objective and reproducible measures of response (or other outcomes) to inform decision making in oncology drug development and could be considered in future studies. In addition, the concordance results add to the body of evidence characterizing a response variable derived from abstracted EHR documentation. Yet, further understanding of the utility of these variables (IRb-RECIST and CAR) in the context of different applications, such as real-world comparator cohorts, or comparative effectiveness studies, will be needed. It may be premature for CAR to be used for direct statistical comparisons with clinical trial RECIST response, but it could provide valuable information to characterize observational outcomes in a population of interest, or to do cross-cohort comparisons using the same variable. It could also be used to generate external context for results from single-arm clinical trials, with the appropriate adjustment methodology to incorporate concordance benchmarks.

Conclusions

We developed an approach to collect and centralize the evaluation of routine-care imaging in patients with mNSCLC (IRb-RECIST), and applied the ensuing readouts to better characterize an EHR-based CAR variable, performing a thorough quantitative assessment across both. This is key in the development of a framework for generating quantitative, objective response measures from routine-care imaging. Prior work[9] indicated that the CAR variable produced clinically meaningful cohort-level results in their correlation with RECIST results and their association with survival. The present patient-level results reinforce the clinical value of deriving response information based on routine imaging, and highlight areas of future improvement (optimizing data completeness and collection, refining standardization of routine documentation practices) as we progress toward reliable and high-quality metrics and end points for observational oncology studies.