Postapproval trials versus patient registries: comparability of advanced melanoma patients with brain metastases.

Postapproval trials and patient registries have their pros and cons in the generation of postapproval data. No direct comparison between clinical outcomes of these data sources currently exists for advanced melanoma patients. We aimed to investigate whether a patient registry can complement or even replace postapproval trials. Postapproval single-arm clinical trial data from the Medicines Evaluation Board and real-world data from the Dutch Melanoma Treatment Registry were used. The study population consisted of advanced melanoma patients with brain metastases treated with targeted therapies (BRAF- or BRAF-MEK inhibitors) in the first line. A Cox hazard regression model and a propensity score matching (PSM) model were used to compare the two patient populations. Compared to patients treated in postapproval trials (n = 467), real-world patients (n = 602) had significantly higher age, higher ECOG performance status, more often ≥3 organ involvement and more symptomatic brain metastases. Lactate dehydrogenase levels were similar between both groups. The unadjusted median overall survival (mOS) in postapproval clinical trial patients was 8.7 (95% CI, 8.1-10.4) months compared to 7.2 (95% CI, 6.5-7.7) months (P < 0.01) in real-world patients. With the Cox hazard regression model, survival was adjusted for prognostic factors, which led to a statistically insignificant difference in mOS for trial and real-world patients of 8.7 (95% CI, 7.9-10.4) months compared to 7.3 (95% CI, 6.3-7.9) months, respectively. The PSM model resulted in 310 matched patients with similar survival (P = 0.9). Clinical outcomes of both data sources were similar. Registries could be a complementary data source to postapproval clinical trials to establish information on clinical outcomes in specific subpopulations.

Introduction

For certain subgroups of patients, long-term benefits, safety and efficacy of novel drugs or drug combinations may not be proven at the time of market access. To obtain additional evidence, specific patient populations are investigated in postapproval clinical trials. These trials use strict inclusion and exclusion criteria, creating efficacy and safety data in a relatively homogenous patient population [1]. A new method to collect postapproval clinical data, incorporating patients in a ‘real-world’ setting, is the use of drug-, patient- or disease-specific registries. To guide the use of these registries in the medicine evaluation by regulators, the European Medicines Agency (EMA) launched a ‘Patient Registries Initiative’ in 2015 [2]. Several programs and projects were also initiated by the US Food and Drug Administration to use real-world evidence in regulatory decision making [3].

Postapproval clinical trials and registries have their pros and cons. Trials can be time-consuming and expensive and are not always representative of the real-world population. Trials include a homogeneous study cohort consisting of patients with similar characteristics and treatments, resulting in high internal validity and more readily interpretable results. Assessment of safety and disease progression in trial patients are structured and similar for all patients. In contrast, real-world patients have heterogeneous patient and disease characteristics and are not always treated identically. Hence, efficacy and safety results are not obtained, assessed and reported in the same way. Data on patients in registries can be an advantage in answering questions related to long-term outcomes and the safety of medicines, which cannot be investigated in postapproval clinical trials with a short follow-up. Furthermore, large registries, covering all patients with a specific disease or disease stage, provide information on higher numbers of specific patient populations, compared to oftentimes limited numbers of patients in postapproval clinical trials. These advantages of registries may be valuable to assess the benefit of cancer therapies. The need for postapproval effectiveness data is becoming increasingly more important since benefit-risk assessments of new medicines have been based on smaller, single-arm trials. Postapproval data can be used for additional insights of these medicines and to confirm expected benefits.

Multiple targeted therapies and immunotherapies received marketing authorization, leading to increased treatment options for stages III and IV (advanced) melanoma patients [4]. Phase III trials, including these drugs, showed significant improvements in the survival of these patients [5-9]. The approved indication of drugs is sometimes broader than the strictly selected patient population investigated in trials. For example, during the time of approval of the BRAF-inhibitors, regulators requested more data of the applicant on the efficacy of the BRAF-inhibitors in advanced melanoma patients with brain metastases. Patients with brain metastases, whether asymptomatic or symptomatic, were excluded from pivotal trials, but the approved indication included these patients [10-12]. Such patients belong to a subgroup of significant interest as melanoma is the third most common cancer with metastases to the brain [13]. Postapproval trials in advanced melanoma patients with brain metastases were conducted to answer questions from regulators on the efficacy of targeted therapies in this particular subgroup.

Data on advanced melanoma patients have also been collected in the nationwide Dutch Melanoma Treatment Registry (DMTR). This disease-specific registry was established in 2013 to assure the safety and quality of advanced melanoma care in the Netherlands by providing insight into the outcomes of daily clinical practice [14]. The DMTR includes all advanced melanoma patients and provides information about patient, tumor and treatment characteristics as well as outcomes. The DMTR is, thus, a potential source for postapproval data collection.

Earlier research has shown that patient registries are a less used resource for regulatory authorities in the assessments of drugs [15]. This may be because, although postapproval clinical trials and patient registries can both be used for postapproval data collection, no direct comparison of patient characteristics and survival in these two postapproval data sources yet exists. This study uses advanced melanoma patient data to compare these two data sources for postapproval data collection and to explore whether the DMTR can complement postapproval clinical trials.

Methods

Patients

The study population consisted of advanced melanoma patients (≥18 years) with symptomatic or asymptomatic brain metastases treated with first-line targeted therapy, including BRAF-MEK combination and BRAF monotherapy. We constructed two treatment groups: patients treated in postapproval clinical trials (trial patients) and patients treated in daily clinical practice (real-world patients). Real-world patients were treated in the 14 designated melanoma centers in the Netherlands between 2012 and 2019.

Data sources

The database of the Medicines Evaluation Board was used, including postapproval clinical trials. These data were supplied by applicants to the EMA after market authorization. The second data source, the DMTR, is a nationwide prospective patient registry, including all advanced melanoma patients diagnosed since 2012 [14].

Pooling of trials

Patient- and tumor baseline characteristics of patients treated in single-arm postapproval trials were analyzed with a chi-square test to find potential differences between these trials. Analyzed baseline characteristics were age, gender, Eastern Cooperative Oncology Group Performance score (ECOG PS), serum lactate dehydrogenase (LDH), prior brain radiation, previous brain surgery, number of organ sites with metastases, symptomatic or asymptomatic brain metastases, type of targeted therapy and year of treatment. These baseline characteristics have been previously described as predictive factors for clinical outcomes of advanced melanoma patients [16]. Overall survival (OS) of the trials was adjusted for prognostic factors by a Cox proportional hazard regression model. After adjustment for baseline characteristics, the hazard ratios for survival did not differ between the trials. The trials could, therefore, be aggregated as one population. The total number of trial patients was compared to real-world patients.

Primary outcome

The primary outcome in the postapproval clinical trials was the (intracranial) response rate. The secondary outcome was OS meeting the requested effectiveness of advanced melanoma patients with brain metastases. Because the intracranial response rate is not registered in the DMTR, we used the Kaplan–Meier method to analyze median OS (mOS) as the primary outcome with corresponding 95% confidence intervals (CI).

Statistical analysis

Patient characteristics of trial and real-world patients were analyzed using descriptive statistics. OS was calculated from the date of start systemic therapy until the date of death from any cause or date of the last contact. Patients who did not reach the endpoint were censored at last contact. Median follow-up time was calculated using the reverse Kaplan–Meier method [17].

Two statistical models were used to compare trial patients with real-world patients. The first was a multivariable Cox hazard regression model. The proportionality assumption of the variables in the Cox model was investigated using scaled Schoenfeld residuals. OS was adjusted for baseline characteristics using the Cox model. The Kaplan–Meier estimates of the two patient populations were compared with a log-rank test. The second model used to compare the two data sources was propensity score matching (PSM). This model gives a propensity score to each individual based on baseline patient, tumor and treatment characteristics. The propensity scores of patients from the trial group are matched to individuals from the real-world data group. This model only focuses on matched patients, meaning that these patients were equal to each other in terms of patient, tumor and treatment characteristics. Matching was performed on age, gender, ECOG PS LDH, number of organ sites, type of therapy, year of treatment and symptomatic or asymptomatic brain metastases. Because matched groups are not independent in this model, the two groups were compared with the stratified log-rank test.

Data handling and statistical analyses were performed using the R software system for statistical computing (version 3.6.1.; packages tidyverse, lubridate, car, survival and survminer).

Results

Baseline characteristics

Four single-arm postapproval clinical trials in patients with advanced melanoma and brain metastases were pooled, resulting in a dataset of 467 patients. All these trial patients were treated with first-line BRAF-MEK combination or BRAF monotherapy. Between 2012 and 2019, 1199 patients with advanced melanoma and brain metastases were registered in the DMTR (Fig. 1), with 602 (50%) patients treated with first-line targeted therapy. The main differences in patient- and tumor characteristics between trial and real-world patients treated with first-line targeted therapy are shown in Table 1. Real-world patients had significantly poorer characteristics: higher age, higher ECOG PS, more organ site involvement and more often symptomatic brain metastases. The unadjusted mOS of real-world patients was 7.2 (95% CI, 6.5–7.7) months and significantly lower (P < 0.01) than the mOS of postapproval clinical trial patients, which was 8.7 (95% CI, 8.1–10.4) months (Fig. 2). After having adjusted for baseline characteristics, median survival times in the two groups were similar.

Fig. 1

Flowchart of patient population and the two statistical models used.

Table 1

Baseline characteristics of targeted therapy treated advanced melanoma patients with brain metastases in post approval clinical trials and in real-world

Baseline	Clinical trials	Real-world	P value
Patients, n	467	602
Gender, n (%)			0.069
Male	295 (63.2)	346 (57.5)
Female	172 (36.8)	256 (42.5)
Age [median (range)]	53 (19–87)	58 (18–92)	<0.001
ECOG PS, n (%)			<0.001
≤1	459 (98.3)	401 (66.6)
≥2	8 (1.7)	129 (21.4)
Unknown	0	72 (12.0)
LDH, n (%)			<0.001
Not determined	0	21 (3.5)
Normal	212 (45.4)	293 (48.8)
1-2 × ULN	148 (31.7)	174 (29.0)
>2 × ULN	107 (22.9)	112 (18.7)
Distant metastases, n (%)			<0.001
<3 organ sites	196 (42.0)	172 (28.6)
≥3 organ sites	271 (58.0)	430 (71.4)
Type of therapy, n (%)			<0.001
BRAFi mono	342 (73.2)	262 (43.5)
BRAFi/MEKi combi	125 (26.8)	340 (56.5)
Brain metastases, n (%)			<0.001
Asymptomatic	426 (91.2)	189 (31.5)
Symptomatic	41 (8.8)	411 (68.5)
Brain surgery, n (%)			<0.001
No	364 (77.9)	560 (93.0)
Yes	103 (22.1)	42 (7.0)
Brain radiation, n (%)			<0.001
No	318 (68.1)	336 (55.8)
Yes	149 (31.9)	265 (44)
Unknown	0	1 (0.2)
Start target therapy year, n (%)			<0.001
2010–2011	219 (46.9)	0
2012–2013–2014	188 (40.3)	185 (30.7)
2015–2016	60 (12.8)	168 (27.9)
2017–2018–2019	0	249 (41.4)

Distant metastases – number of organ sites with metastases.

ECOG PS, Eastern Cooperative Oncology Group Performance score; LDH, lactate dehydrogenase.

Fig. 2

Unadjusted survival of trial- and real-world patients. Log-rank test comparison of unadjusted overall survival of patients treated in postapproval clinical trials with patients treated in the real-world. mo, months; CI, Confidence interval; OS, overall survival.

Cox hazard regression model

A multivariable Cox regression hazard model, comparing real-world patients to those treated in the postapproval clinical trials, showed a hazard ratio on survival of 1.19 (95% CI, 0.93–1.51; P = 0.165) for real-world patients. This hazard ratio is adjusted for the prognostic factors shown in Fig. 3. In this Cox model, age >70 years, ECOG PS ≥2, symptomatic brain metastases, metastases in ≥3 organ sites and elevated LDH were significantly negatively associated with survival. As compared to BRAF monotherapy, combination therapy with BRAF-MEK was significantly positively associated with survival (hazard ratio, 0.67; 95% CI,0.54–0.84; P < 0.001). Another factor that improved survival was prior brain surgery, compared to patients who did not receive brain surgery, with an hazard ratio of 0.68 (95% CI, 0.53–0.87; P = 0.002). Survival was also influenced by the start year of targeted therapy, with patients treated in more recent years having an improved survival (Fig. 3). The OS of the two subgroups was adjusted for prognostic factors, which led to a statistically insignificant difference in mOS of, respectively, 8.7 (95% CI, 7.9–10.4) compared to 7.3 (95% CI, 6.3–7.9) months for trial and real-world patients.

Fig. 3

Hazard ratios of a multivariable Cox proportional hazard model of advanced melanoma patients with brain metastases treated in postapproval clinical trials and the real world. CI, confidence interval; ECOG, Eastern Cooperative Oncology Group; LDH, lactate dehydrogenase.

Propensity score matching model

The total patient population consisted of 602 real-world patients and 467 trial patients. To perform PSM, 76 real-world patients with missing values were excluded from the model before matching. In total 993 patients remained for matching and 310 (30.6%) patients were matched, resulting in 155 pairs of patients from the real-world and trial population. After matching, none of the prognostic factors differed between the two data sources. The stratified log-rank test resulted in similar survival (P = 0.9) of real-world and trial patients (Fig. 4). The other 683 patients could not be matched with an individual from the other data source, because of differences in ECOG PS and the presence of symptomatic versus asymptomatic brain metastases. Only a limited number of patients with symptomatic brain metastases and ECOG PS ≥2 were included in the trials (Table 1).

Fig. 4

Overall survival of matched advanced melanoma patients with brain metastases treated in targeted therapy postapproval clinical trials and the real-world after propensity score matching.

Discussion

Comparison of the outcomes of the data sources

In this study, we compared outcomes of postapproval clinical trials with data from a disease-specific registry to explore whether a registry can complement postapproval clinical trials after market authorization of medicines. The absolute difference in the unadjusted median OS of the two groups was minimal (1.5 months), which can be explained by the fact that these patients with brain metastases are included in postapproval clinical trials, but would have been excluded from the phase-III trials. Therefore, these patients included in the postapproval clinical trials are more similar to real-world patients than advanced melanoma patients included in phase-III trials. However, postapproval trial patients did not completely represent the real-world population. There were still major differences in patient and tumor characteristics between the two groups. Real-world patients had more often symptomatic brain metastases (69%) compared to trial patients (9%). Differences were also found in age, ECOG PS, organ site involvement and whether prior brain surgery was performed. Patients receiving brain surgery have relatively more favorable characteristics. These differences resulted in seemingly better survival among clinical trial patients, which is in general, the case for other conditions as well. After correction for baseline characteristics, the clinical outcomes of the two patient populations did not differ. We can, therefore, argue that no other prognostic factors (i.e. stricter treatment schedules and controls) contribute to the differences in survival between these patients treated in postapproval trials and real-world patients registered in the DMTR. The survival of real-world patients who were matched to trial patients was similar.

Other outcomes for measurement

The primary outcome in the postapproval trials was intracranial response. This outcome has not been registered in the DMTR and, therefore, the OS of the two patient groups was compared. Survival is an objective outcome that is well documented in both sources.

Safety or quality-of-life of advanced melanoma patients with brain metastases were not investigated and compared in this study because registration of adverse events in postapproval trials and the DMTR are different. Individual data on the grade of toxicity in the postapproval trials were lacking, and because only grade 3 and 4 toxicities are registered in the DMTR, we could not compare safety. Furthermore, the patient-reported outcome measurements questionnaires are filled out by a relatively low number of patients in the DMTR and were not used in these postapproval trials. In the future, safety and quality-of-life could also be measured in registries using similar methods as in trials, including grade 1 and 2 toxicities and more specific data on the date of toxicity and the consequences of toxicity (discontinuation and hospitalization). Registries can then be used for safety concerns or measurements in postapproval data collection.

Requirements of the data sources

Postapproval clinical trials and registries are both used by regulators in the medicine’s evaluation. However, earlier research showed that registry data are more often used by regulators for (long-term) safety of medicines and less used for effectiveness [18]. In only seven of the 73 (9.6%) registries, that were used in the risk management plan of medicines, the primary goal was real-world safety and effectiveness [18]. Real-world information on medicines is also used by other stakeholders, such as health technology assessment (HTA) organizations, payers and manufacturers [19].

Both data sources need to meet several requirements to lead to valuable and trustworthy data for regulatory assessments. In both data sources, patient numbers and the length of follow-up should be of sufficient duration to measure clinical outcomes. The proportion of excluded patients in postapproval trials should be minimalized to be representative of actual clinical care [1]. The postapproval trials used in this study were single-arm trials and therefore lacked a comparator. The inclusion and exclusion criteria used, except for the presence of brain metastases, were similar to those used in the pivotal clinical trials, leading to an underrepresentation of the real-world population.

To reduce bias, registries need to be validated, complete and consistent. This requires consistency in the registration of variables by hospitals and the measurement of clinical outcomes. Important factors supporting the consistency, allowing the use of registries in medicine evaluation by regulators, would include, for example, the use of common datasets and coding terminologies, complete data collection, and quality assurance and consistent governance [15,20]. The registry used in this study, the DMTR, is a validated, nationwide patient registry with limited missing patients and data. DMTR data quality control is performed by medical oncologists and independent reviewers. Data managers are trained to register data accurately. They follow patients until death, and electronic patient records are checked every 3 months. DMTR data are also accessible and shared with multiple stakeholders [14]. At the same time, because an adequate (untreated) comparator is lacking, registry data are not sufficient to determine the benefit/risk balance of novel drugs. Furthermore, registry data reflect the less tightly controlled and registered dosing schedules and medication adherence of real-world practice, possibly leading to variation in clinical outcomes.

Deciding which data source to use

The comparison of data sources leads to the question of which data source regulators should request and allow when postapproval data is needed. This choice highly depends on three main aspects. First, it depends on the questions that regulators have when they are deciding on marketing approval. When researching clinical benefit or treatment strategies of novel drugs, trials are preferred. Registries can be of major value to indicate outcomes of specific populations, such as mucosal or uveal melanoma patients and ineligible patients [21]. Data from registries can also be used to measure outcomes of practical treatment schedules and strategies, such as treatment steps and treatment duration. These results from registries can then be confirmed by randomized controlled trials. An example of supporting evidence from DMTR data on advanced melanoma patients treated with first-line targeted therapy with high LDH showed subsequent immunotherapy could lead to long-term survival if normalization of LDH was reached [22]. This information is easily obtained with the use of a registry including real-world practices.

Second, the choice for a data source depends on the availability of a high-quality registry and available data, or the resources to initiate postapproval clinical trials. In the case of the treatment with targeted therapies in patients with brain metastases, the intracranial response was used as a surrogate endpoint to assess effectiveness in these patients. An intracranial response in these patients, regulators argued, would lead to a clinical benefit, meaning treatment of these patients would be justified. Because the intracranial response is not registered in the DMTR, the core data set should have been expanded to match regulatory needs. The addition of such a data key point in a registry would in general require less effort than conducting new postapproval studies.

Third, the choice depends on the patient population in question. Registries are a better data source when the remaining questions concern patients rarely included in phase III trials, such as the elderly or children. Registries can include rare patient populations over a longer period of time, leading to more data. This was the case when the Pediatric Committee of the EMA requested additional safety information in pediatric patients treated with ipilimumab [23]. The DMTR was chosen as a data source for these data because the number of children with advanced melanoma is limited. For this patient population, conducting a postapproval trial would be very challenging.

Broader perspective

Setting up or expanding disease-specific registries for postapproval evidence requires criteria sets. Registries come with a registration burden for caregivers and, therefore, the data set should be minimalized, including only the data points most important for the multiple stakeholders (not only regulators but also caregivers, patients, HTAs, insurers and pharmaceutical companies). Such concision would reduce the registration burden. To use registry data to evaluate medicines, it also needs to be systemically collected, using criteria similar to those used in trials. This means data should also be noted concisely in electronic patient records or that key data points should be automatically filled. Using registries in this way to gather postapproval data on medicines will possibly be more effective than setting up and conducting (multiple) trials, which may have a longer lead-time. In a study including 600 nonrequired postapproval trials, the median duration of these trials was 37 (22–57) months [24]. The median duration of trials on cancer or hematology (n = 437) was 43 (29–66) months. Of 204 completed or terminated postapproval trials, the duration from completion to reporting was 16 (13–25) months. This research [24] also showed 32% of the postapproval trials did not report results within 35 months after trial completion. The speed of data collection and gathering depends on the research question and the availability of information in a registry. Delays affect patient care but can be addressed by registries from which outcomes could eventually be generated and reported more quickly if the data are already available.

Conclusion

High-quality population-based registries could be a complementary data source to postapproval clinical trials to establish information on clinical outcomes in specific subpopulations with advanced melanoma after market authorization. Disease registries are more representative for the real-world population than patients treated in postapproval clinical trials, leading to improved understanding of the effectiveness of medicines in the real world. Postapproval data from registries can support regulatory decisions for remaining questions on new medicines instead of postapproval studies.

Acknowledgements

For the Dutch Melanoma Treatment Registry (DMTR), the Dutch Institute for Clinical Auditing foundation received a start-up grant from governmental organization The Netherlands Organization for Health Research and Development (ZonMW, grant number 836002002). The DMTR is structurally funded by Bristol-Myers Squibb, Merck Sharpe & Dohme, Novartis and Roche Pharma. Roche Pharma stopped and Pierre Fabre started funding of the DMTR in 2019. For this work, no funding was granted.

Conflicts of interest

A.v.d.E. has advisory relationships with Amgen, Bristol-Myers Squibb, Roche, Novartis, MSD, Pierre Fabre, Sanofi, Pfizer, Ipsen and Merck and has received research study grants not related to this article from Sanofi, Roche, Bristol-Myers Squibb, Idera and TEVA and has received travel expenses from MSD Oncology, Roche, Pfizer and Sanofi and has received speaker honoraria from BMS and Novartis. M.B.S. has consultancy/advisory relationships with Pierre Fabre, MSD and Novartis. J.d.G has consultancy/advisory relationships with Bristol-Myers Squibb, Pierre Fabre, Servier, MSD and Novartis. G.H. consultancy/advisory relationships with Amgen, Bristol-Myers Squibb, Roche, MSD, Pfizer, Novartis and Pierre Fabre and has received research grants not related to this paper from Bristol-Myers Squibb, Seerave. E.K. has consultancy/advisory relationships with Bristol-Myers Squibb, Novartis, Merck and Pierre Fabre, and received research grants not related to this paper from Bristol-Myers Squibb. K.S. has advisory relationships with Bristol-Myers Squibb, Novartis, MSD and Pierre Fabre and received honoraria from Novartis, MSD and Roche. A.v.d.V. has consultancy relationships with Bristol-Myers Squibb, MSD, Roche, Novartis, Pierre Fabre, Pfizer, Sanofi, Ipsen, Eisai and Merck. J.H. has advisory relationships with Aimm, Achilles Therapeutics, Amgen, AstraZeneca, Bayer, Bristol-Myers Squibb, Celsius Therapeutics, GSK, Immunocore, Ipsen, MSD, Merck Serono, Novartis, Neogene Thereapeutics, Neon Therapeutics, Pfizer, Roche/Genentech, Sanofi, Seattle Genetics, Third Rock Ventures and Vaximm and has received research grants not related to this article from Bristol-Myers Squibb, MSD, Neon Therapeutics and Novartis. All grants were paid to the institutions. The funders had no role in the writing of this article or decision to submit it for publication. For the remaining authors, there are no conflicts of interests.