Impact of early surveillance on safety signal identification in the CathPCI DELTA study.

OBJECTIVES: The CathPCI Data Extraction and Longitudinal Trend Analysis study was designed to determine the feasibility of conducting prospective surveillance of a large national registry to perform comparative safety analyses of medical devices. We sought to determine whether the complementary use of retrospective case data could improve safety signal detection time.
DESIGN: We performed a simulated surveillance study of the comparative safety of the Mynx vascular closure device (VCD) with propensity score matched alternate VCD recipients, using both retrospective and prospective cohort data.
SETTING: Centers within the USA using the National Cardiovascular Data Registry (NCDR) CathPCI Registry. PARTICIPANTS: Percutaneous coronary intervention cases captured within the NCDR CathPCI Registry from July 1, 2009 to September 30, 2013 were included in the analysis.
INTERVENTIONS: None. MAIN OUTCOME MEASURES: Absolute and relative risk (RR) of any vascular complication (a composite of bleeding at access site, hematoma at access site, retroperitoneal bleeding, and other vascular complications requiring treatment); time to signal detection.
RESULTS: A safety alert was detected for the primary outcome of "any vascular complication" after 15 months of surveillance and was sustained for the study duration (absolute risk of any vascular complication, 1.20% vs 0.73%, RR, 1.63; 95% CI 1.50 to 1.79; p<0.001). The safety signal was identified 12 months earlier with the use of retrospective case data than during the initial study.
CONCLUSIONS: Prospective, active surveillance of cardiovascular registries is feasible to perform comparative analyses of medical devices. Retrospective data may complement prospective surveillance to improve time to signal detection, indicating the need for earlier prospective application of safety surveillance for devices new to the market. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Current strategies for post-market safety evaluations of medical devices are insufficient to ensure public safety. Prospective surveillance of medical device safety leveraging high quality clinical data repositories may address several of these limitations.

Using both prospectively and retrospectively collected data within high quality registries may improve the time to safety alert identification in safety surveillance studies of medical devices.

Efficient and reliable identification of safety differences among medical devices may support regulatory decision making, iterative improvements, and educational initiatives surrounding new medical devices. Active surveillance tools, such as Data Extraction and Longitudinal Trend Analysis, may be leveraged.

Background

Assuring public safety after medical device approval and widespread adoption is critically important. However, current pre-market safety evaluations are constrained by small sample size and short duration of follow-up, limiting the ability to detect low frequency safety events over the life-cycle of a medical device. Additionally, post-market safety evaluations rely heavily on voluntary reporting of safety events, leading to incomplete ascertainment of safety events.1–10 Prospective, active surveillance of large, high quality clinical data repositories addresses these limitations, and has been identified as a strategic priority by the Food and Drug Administration. We developed a set of active surveillance tools, referred to as DELTA (Data Extraction and Longitudinal Trend Analysis system), to support near real-time safety monitoring. The methods and informatics infrastructure of DELTA have been previously described and validated.11–14

The CathPCI DELTA study was conducted to evaluate the feasibility of prospective, active surveillance of a National Cardiovascular Data Registry (NCDR) for assessment of the post-market safety of implantable medical devices. We assessed the relative safety of a vascular closure device (VCD) (Mynx, Cardinal Health) using data from the NCDR CathPCI Registry, and identified a sustained safety alert after 9 months of surveillance.15 However, it is unknown whether a safety alert associated with this device could have been detected earlier in the device life-cycle. We therefore sought to explore whether earlier safety signal detection was feasible using complementary prospective and retrospectively identified Mynx VCD cases within the CathPCI Registry.

Methods

Study design and oversight

A simulated prospective surveillance study of the comparative safety of the identified VCD using both retrospective and prospective data was designed, anticipating improvement in safety signal detection time. The study was completed using data from the NCDR CathPCI Registry. The NCDR CathPCI Registry is cosponsored by the American College of Cardiology and Society for Cardiovascular Angiography and Interventions, and is a large, contemporary national registry of patients undergoing cardiac catheterization procedures and/or percutaneous coronary intervention (PCI). The database included more than 15,000 hospitals and capture ~90% of PCI procedures. Internal quality assurance protocols and quality checks, as well data auditing are conducted to ensure data completeness, validity, and reliability.

For this study, patient-level data from the CathPCI Registry were fully deidentified in accordance with recommendations of the Health Insurance Portability and Accountability Act and the Office of the National Coordinator for Health Information Technology prior to being uploaded to DELTA. Analysis parameters were retained and applied to the expanded dataset.

Patient eligibility, device exposures, and endpoint definitions

We identified all patients in the CathPCI Registry aged ≥18 years who received treatment with VCD after PCI performed with femoral arterial access from January 1, 2011 to September 30, 2013 for the initial (prospective only) analysis. For this extended study, cases in the CathPCI PCI beginning July 1, 2009, representing the initiation of the CathPCI version 4 data collection tool, were added to the original data set. Cases of diagnostic cardiac catheterization without subsequent PCI were excluded. Additionally, patients who received treatment with an intra-aortic balloon pump, ventricular assist device, those who were treated with non-femoral arterial access, multiple VCDs, who had more than one arterial access site, and those treated with VCDs that did not include an implantable component were excluded from the final analysis. VCDs with identical mechanism of action and implantable components were grouped together into “device families” to increase analytic power (online supplemental appendix table 1).

All endpoints and covariates were defined according to the CathPCI Registry, version 4.4 definitions. The primary study endpoint was “any vascular complication,” a composite of access site bleeding requiring treatment, access site hematoma requiring bleeding, retroperitoneal bleeding, or any vascular complication requiring additional intervention. “Other vascular complications requiring treatment” could include, but were not limited to, access site occlusions, peripheral embolizations, dissections, pseudoaneurysms and/or arterial-venous fistulas. Any noted vascular complication must have had an intervention such as a fibrin injection, angioplasty, or surgical repair to qualify.

Prespecified secondary endpoints included access site bleeding requiring treatment and post-procedure blood transfusion. Outcomes were analyzed until the time of hospital discharge, as limited by the data available within the CathPCI Registry.

Outcomes were analyzed until the time of hospital discharge, as limited by the data available within the CathPCI Registry.

Surveillance and statistical methods

Active surveillance system

The analysis was performed using the DELTA suite of active surveillance tools. DELTA is an open-source collection of integrated software components linking statistical analytic tools with data repositories. DELTA has been previously validated for prospective monitoring of clinical registries and data sets to support risk-adjusted prospective safety surveillance analyses.

Data were delivered to DELTA on a predetermined schedule of updates, and the cumulative safety analysis was automatically regenerated within DELTA.

Propensity score matching and event rate estimation

We developed multivariable adjusted logistic regression models to estimate the probability of being treated with the Mynx or an alternate VCD of interest, conditional on the included covariates. The model included previously identified risk factors for the adverse outcomes of interest, as well as factors considered to influence the selection of a specific device. A total of 15 variables were included in the final propensity score model. All covariates were defined in accordance with the NCDR CathPCI Coder’s Data Dictionary, versions 4.4 (online supplemental appendix). Demographic and comorbid variables included age, body mass index, sex, diabetes, history of chronic lung disease, hypertension, pre-procedure creatinine, and history of peripheral artery disease. Variables related to clinical presentation and angiographic findings in the final propensity score model included emergent cardiac catheterization, non-ST elevation myocardial infarction on presentation, bivalirudin exposure, left main coronary artery PCI, number of coronary vessels treated during index presentation, fluoroscopy time, and total number of PCI procedures during admission.

A propensity score matched control population was identified from the population of patients treated with non-Mynx family, alternative VCDs. The propensity matched comparison group was selected on the basis of a non-parsimonious propensity model. Matched controls were selected in a 1:1 ratio, within 6 months of the date of the case device implant/use, using a fixed propensity probability caliper width of 0.01 using a greedy matching algorithm.7 10 16 17 At each quarterly data upload, the DELTA system re-matched the case sets and adverse event rate were calculated.

Missing data were handled using univariate rules, assuming absence of a condition for dichotomous variables, and using the median value for continuous variables.18 Absolute standardized difference (per cent) in covariate proportions and means were calculated to assess the relative imbalance between the exposed (Mynx VCD recipients) and unexposed (alternate VCD) groups with values greater than 10% considered severely imbalanced.

Adverse event rates were calculated quarterly for the propensity score matched cohorts in each analysis. Safety alerts were triggered if the CIs around the differences between two independent proportions, as measured by the Wilson method, did not cross zero, indicating a statistically significant difference between the exposed and control cohorts.19 CIs were corrected for multiple comparisons through use of the adapted O’Brien-Fleming method.20

Results

The final study population included 109,857 cases of Mynx device recipients from 1479 centers between July 1, 2009 and September 30, 2013. Compared with the initial cohort, the study population included 36,693 additional earlier Mynx cases, with an additional 18 months of surveillance time compared with the initial analysis. Propensity score matching resulted in 100% of Mynx cases being matched to alternate VCD cases, with adequate distribution of risk factors as evidenced by post-matching absolute standardized differences less than 0.10 (table 1).

Table 1

Population covariate comparison (pre-match and post-match)

Covariate	Total study population			After propensity match
	Mynx VCD	Alternate VCD	Std. Diff.	Mynx VCD	Alternate VCD	Std. Diff.
	(n=109,857)	(n=925,355)		(n=109,857)	(n=109,857)
Age (years)	65.19±11.90	64.95±12.09	0.0200	65.19±11.90	65.22±11.89	0.0026
Female gender	34.12%	30.49%	0.0777	34.12%	34.47%	0.0072
Body mass index (kg/m2)
<21	3.52%	3.67%	0.0080	3.52%	3.55%	0.0014
≥25 and <30	35.7%	37.0%	0.0282	35.7%	35.7%	0.0003
≥30	45.31%	43.21%	0.0423	45.31%	45.38%	0.0014
Diabetes	39.1%	35.3%	0.0795	39.1%	39.5%	0.0084
Chronic lung disease	16.3%	13.5%	0.0783	16.3%	16.3%	0.0022
Hypertension	84.7%	80.9%	0.1001	84.7%	84.8%	0.0037
Baseline creatinine (mg/dL)	1.19±1.03	1.16±0.95	0.0627	1.19±1.03	1.20±1.06	0.0095
Peripheral arterial disease	12.7%	9.7%	0.0964	12.7%	12.9%	0.0047
Emergent procedure status	11.7%	17.7%	0.1693	11.7%	11.8%	0.0018
NSTEMI on presentation	17.17%	19.25%	0.0540	17.17%	17.01%	0.0040
Bivalirudin exposure	66.9%	61.8%	0.1062	66.9%	67.5%	0.0119
Left main coronary artery PCI	2.06%	2.11%	0.0037	2.06%	2.00%	0.0043
Number of vessels treated during index PCI	1.42±0.70	1.44±0.73	0.0292	1.42±0.70	1.42±0.71	0.0049
Fluoroscopy time (min)	12.33±9.77	14.04±11.15	0.1340	12.33±9.77	12.46±9.42	0.0102
Total number of PCI during admission	1.0481	1.0467	0.0062	1.0481	1.0483	0.0006

Plus–minus values are means±SD. The standardised difference (the mean between-group difference divided by the SD) and was calculated to assess the relative imbalance between the exposed and unexposed groups, with values of less than 0.100 considered to be adequately balanced. The body mass index is the weight in kilograms divided by the square of the height in meters.

CAD, coronary artery disease; NSTEMI, non-ST elevation myocardial infarction; PCI, percutaneous coronary intervention; Std. Diff., absolute percentile standardized difference; VCD, vascular closure device.

After 15 months of surveillance, a safety alert for was detected for the primary outcome of “any vascular complication” during the third calendar quarter of 2010 and was sustained for the study duration (absolute risk of any vascular complication, 1.20% vs 0.73%, relative risk (RR), 1.63; 95% CI 1.50 to 1.79; p<0.001) (figure 1 and table 2).

Figure 1

Cumulative incidence of vascular complications among recipients of the Mynx vascular closure device (VCD). Shown is the cumulative incidence of vascular complications after being treated with the Mynx VCD or a propensity matched alternative device. Green circles indicate no significant difference in adverse event rates between Mynx VCD and a propensity matched alternate device. Red circles indicate higher than expected rates of adverse events that trigger a safety alert. Black squares indicate the observed rate of adverse events in the propensity matched alternate device group. Solid black lines represent 95% CIs, with dashed black lines representing 95% CIs after adjustment for multiple comparisons. The solid green line represents the cumulative number of exposures to the Mynx VCD. Q1 through Q17 represent each calendar quarter (3 months) of data analysis. DELTA, Data Extraction and Longitudinal Trend Analysis.

Table 2

Event rate and time to alert by population group

All patients	Mynx(N=109,857) (%)	Alternate device(N=109,857) (%)	Relative risk (95% CI)	Time to alert	Improvement in time to alert
Vascular complications	1.20	0.73	1.63 (1.50 to 1.79)	15 months	12 months
Access-site bleeding	0.37	0.26	1.43 (1.23 to 1.67)	21 months	27 months
Blood transfusion	1.73	1.45	1.19 (1.12 to 1.28)	24 months	9 months

Compared with the initial analysis which did not include retrospective cases, the safety signal was identified 12 months earlier. During this interval, 24,949 Mynx implants following PCI were captured in the CathPCI Registry.

For the outcome of post-procedure access site bleeding, a safety alert was identified 18 months after study initiation (absolute risk 0.37% vs 0.26%, RR 1.43, 95% CI 1.23 to 1.67), 27 months earlier compared with the initial analysis. For the outcomes of transfusion, a safety alert was identified 21 months after study initiation (absolute risk 1.73% vs 1.45%, RR 1.19, 95% CI 1.12 to 1.28), 9 months earlier compared with the initial analysis. Among prespecified high-risk subgroups including age >70, female sex, and those with diabetes, safety alerts were generated for the primary outcome of “any vascular complication” as well as the secondary endpoints of access site bleeding and need for transfusion (table 2). For all the outcomes, and among each of the subgroups, safety alerts were generated earlier in the current analysis compared with the initial cohort.

Discussion

The CathPCI DELTA study was designed to assess the feasibility of prospective, active safety surveillance of a large clinical data repository to support safety outcomes monitoring of post-market medical devices. We performed a large, propensity score matched, safety surveillance study of a national cardiovascular clinical registry and identified a safety alert for vascular complications associated with use of the Mynx VCD. In the current analysis, we were able to improve safety signal detection time by 12 months through inclusion of 36,693 retrospective Mynx VCD cases over an additional analysis period of 18 months. This analysis confirmed a higher risk of vascular complications, access site bleeding, and transfusion requirement among a larger cohort of Mynx VCD recipients compared with those treated with alternate VCDs.

The RR for vascular complications associated with the Mynx VCD was 1.63 (95% CI 1.50 to 1.79; p<0.001) with an absolute event rate of 1.20% versus 0.73% with alternate devices. Had the device of interest been recalled after identifying the first safety alert in this study, between 952 and 1035 additional vascular complications may have been avoided.

The appropriate actions following detection of a safety alert have not been fully established. Given the large sample sizes available in national data registries, alerts that achieve statistical significance should be interpreted in the context of their clinical significance. Depending on estimations of absolute event rates, risk thresholds that prompt additional action should be prespecified. The regulatory actions that follow should depend not only on risk estimates, but on the gravity of the clinical outcome event of interest, and the confidence in this signal. For example, a safety signal indicating higher rates of death associated with the use of a medical device may prompt more immediate action at a lower risk ratio with lower confidence. The specific actions that follow should integrate several variables including the frequency of events, seriousness of safety events, and necessity of using the device in consideration. Actions might be limited to manufacturers simply considering device modifications or improving education regarding the use of a specific device or may extend to withdrawal or recall of a device if serious safety concerns are demonstrated.

This study specifically focused on safety of arterial VCDs which were initially brought to market in 1996 in the USA to support safe and expedient hemostasis following femoral arterial access for PCI. Several devices are currently available, and differ with respect to the mechanical, pharmacological, and biomaterial components used. Most have been approved through small pre-market studies, and no data currently supports an improvement in rates of post-procedural bleeding or other complications with their use.21 However, given the improvement in patient comfort compared with prolonged manual compression, and significant improvement in time to achieve hemostasis, VCDs remain widely available and are commonly used. While VCDs offer a convenient solution for arterial hemostasis, the use of such devices, and new devices, should be held to rigorous standards with little tolerance for excess in adverse event rates given their non-essential role in most PCI procedures.

This study has several important limitations. Safety signals, once identified, should be interpreted with caution. We performed robust risk adjustment through the use of propensity score matching. However, the possibility of residual confounding cannot be excluded. Additionally, the variables included in the propensity score matched analysis are limited to those available in the CathPCI data collection tool. Additional anatomic variables such as arterial puncture site, which may contribute to residual confounding, were not available in the CathPCI Registry. Finally, study endpoints were limited to in-hospital events available in the Cath PCI registry.

This study demonstrates the feasibility of automated surveillance within a large, high quality data sources to support post-market monitoring of medical devices. Among existing medical devices, using prospective active surveillance methods, augmented by retrospective registry data may improve early identification of safety differences. For novel medical devices, surveillance should begin early in the device life-cycle to allow rapid and actionable safety alert identification. Overall, our analysis demonstrates that prospective, active surveillance within a large clinical registry is feasible to detect safety differences among commonly used cardiovascular devices.