Evaluation of intervertebral body implant performance using active surveillance of electronic health records.

Objectives: To assess the feasibility of using electronic health record (EHR) derived clinical data within an active surveillance setting to evaluate the safety of a novel intervertebral body implant (IVBI) stabilization device. Design: Retrospective, longitudinal observational cohort study comparing clinical outcomes for patients seen through 1 year following spinal fusion surgery. Setting: Lahey Health network, which includes academic tertiary hospitals, outpatient clinics, and independent provider offices in the New England region of the USA. Participants: All spine surgery patients aged 18 or older who underwent thoracic or lumbar spinal arthrodesis surgeries were included. Main outcome measures: The clinical outcomes of patients treated with the CONCORDE Bullet (CB) interbody spine system (DePuy) between April 2015 and December 2018 were compared with those patients receiving alternative spine stabilization interbody device implants. The primary endpoint was reoperation rate at 1 year, with secondary endpoints including the requirement for blood transfusion during index hospitalization, 1 year rate of any cause hospitalization, 1 year rate of surgical site infection, and mortality at 1 year.
Results: Among the 606 patients undergoing thoracic or lumbar spinal fusion surgery during the study period, 136 received only the CB. In comparison with patients who did not receive the CB, no significant differences were found in the rate of reoperation at 1 year or the rates of secondary safety outcomes. Conclusions: Data derived from the EHR can be successfully leveraged to assess the safety of IVBI devices, in this case demonstrating no significant differences in the rates of risk-adjusted safety endpoints between patients undergoing spinal surgery with the CB as compared with alternative spinal implants. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Small studies have suggested there is comparable safety and efficacy between approved intervertebral body implant devices, and real-world data may provide additional information to support post-market evaluation of such devices.

Real-world data sets derived from routinely collected electronic health records can be analyzed using active surveillance techniques implemented via the DELTA (Data Extraction and Longitudinal Trend Analysis) software system to monitor the safety and comparative outcomes of spinal implant devices.

Our study demonstrates the feasibility of applying active surveillance tools to real-world data to evaluate the performance of spinal implants, and may serve as a model for future post-market evaluations of high risk, implanted medical devices.

Introduction

Intervertebral body stabilization device implants (IVBI) have been used with increasing frequency in an effort to achieve more predictable surgical results during spinal fusion surgeries.1 2 The high costs of spinal fusion implants along with limited effectiveness data highlight the importance of developing data collection systems to address the safety and comparative effectiveness of interbody fusion implants. However, like most medical devices, confirmation of the safety and effectiveness of IVBI have been hampered by the small size of pre-market studies and the inadequacy of voluntary reporting of adverse events through the submission of medical device adverse event reports to the Food and Drug Administration (FDA).3–5

Active surveillance of electronic health record (EHR) derived clinical data has been proposed as a strategy that is complementary to the existing pre-market and post-market evaluation tools, which have primarily relied on the evaluation of adverse event reports.6 We have previously developed and validated an open-source, active surveillance software system called DELTA (Data Extraction and Longitudinal Trend Analysis) to leverage data from high quality clinical data sources for the purposes of comparative safety assessment for a variety of high-risk medical devices.7–9 However, to date, DELTA has not been applied to EHR-derived clinical data for the purpose of active surveillance by automated analysis and re-analysis of accumulating clinical data.

To foster the development of optimized post-approval evaluation of medical devices, the FDA founded the National Evaluation System for health Technologies coordinating center (NESTcc) in 2018.10 Johnson & Johnson proposed a safety surveillance study for the CONCORDE Bullet (CB) IVBI in the first series of NESTcc test case studies in an effort to validate the strategy of applying clinical surveillance methods to EHR-derived data. In addition, the sponsor sought high quality clinical information for submission to European Union regulators in order to meet recent changes in requirements for medical device post-market monitoring,11 but importantly, there was no data suggesting a safety concern with the CB device prior to the initiation of the study.

The CB DELTA study was therefore designed to evaluate the use of active surveillance tools applied to EHR-derived data, from a single healthcare system, to assess the safety of a commonly used IVBI used in posterior spinal fusion procedures.

Methods

Study design and oversight

A comprehensive written protocol was developed, and plans for interim data reviews and a study oversight committee were established with representation from NESTcc, PEDSnet, Johnson & Johnson, and Lahey Medical Center, which reviewed and approved the final study protocol prior to any data analysis. The institutional review boards of Lahey Medical Center and Children’s Hospital of Philadelphia reviewed and cleared the study protocol prior to the review of any study data.

Study environment, data source and data element extraction

Epic Systems (Verona, Wisconsin, USA) has been used as the EHR system at all Lahey Health hospitals and clinics to support all clinical activities. Data elements were extracted from Lahey’s clinical data warehouse derived from the Epic EHR, containing demographic, clinical, laboratory, and claims data for all patient encounters in any Lahey facility. The universal device identifier (UDI) system12 was not implemented within the Lahey EHR until after the study period, and therefore device implants were identified through manufacturer model numbers recorded in the EHR surgical log. Of note, all patients undergoing spine surgery at Lahey are routinely followed with in-person or telehealth visits at 30 days and 12 months postoperatively, thereby minimizing the risk of the loss to follow-up in the first year after spine surgery at the medical center.

Patient eligibility, device exposures, and endpoint definitions

Patients 18 years or older, undergoing spinal arthrodesis surgery between April 1, 2015, and December 31, 2018, at a Lahey Health System hospital were included in the study (see online supplemental figure S1). Patients were excluded from the analysis if their index surgical procedure included the cervical spine or if the patient had any spine surgery performed in the 12 months prior to their qualifying surgery.

Among the patients eligible for inclusion, treatment with the device of interest (the CB), as well as prespecified alternative devices (the DePuy Synthes OPAL and the Medtronic CAPSTONE PEEK) was identified by matching surgical log device implant records to device model numbers available from the manufactures.

The primary safety outcome was the proportion of patients undergoing spine reoperation for any cause at 1 year. Secondary outcomes included mortality, hospitalization for surgical site infection, any hospitalization within 1 year, and requirement for blood transfusion during index hospitalization. The safety endpoints and clinical covariate definitions used for this study are provided in online supplemental tables S1 and S2 of the Supplemental content.

Covariate and data validation

Each demographic, clinical, and procedural covariate and outcome was validated through manual chart review of a random 5% sample of patients included in each cohort. Discrepancies found in manual chart review were used to refine covariate filter definitions, and the random 5% manual chart review was repeated until there was 100% agreement between the extraction filters and domain expert chart review for all covariates and outcomes.

Risk adjustment methods

Multivariable adjusted logistic regression models were developed to estimate the probability of being treated with the CB. The model included risk factors for the adverse outcomes of interest, as well as factors considered to influence the selection of IVBI. A total of 12 demographic, clinical and procedural variables were included in the final propensity score model (variable definitions available in online supplemental table S6 of the Supplemental content) including age, gender, preoperative body mass index, history of coronary artery disease, heart failure, diabetes, active smoking and history of any prior spine surgery. Procedural covariates included surgeon specific annual spine surgical volume, emergency surgery status, American Society of Anesthesiology physical status classification, and number of spinal levels stabilized during index surgery.

The propensity-matched comparison group was selected on the basis of a non-parsimonious propensity model and matched in a 1:1 manner with CB cases matched within a caliper width of 0.6 of the SD of the logit of the propensity score.13 Missing data were handled using univariate rules, assuming absence of a condition for dichotomous variables, and using the median value for continuous variables. The relative imbalance between the CB and comparator groups was assessed using absolute standardized mean difference in covariate means and proportions, with values greater than 0.10 considered suboptimally balanced.14

Because the number of patients successfully matched could be low and lead to an underpowered analysis, we also performed inverse probability of treatment weighting (IPTW) analysis. IPTW included all patients in both the treatment and control groups, weighted based on the probability of treatment with the device of interest, and therefore minimizes information loss through case exclusion. Weights were trimmed at 5% and 95% to avoid excessive influence of patients with extremes of propensity scores.15

A ‘Falsification Hypothesis’ analysis was also prespecified in order to assess the possibility of significant residual confounding after propensity matching or weighting. For this analysis, patients were evaluated for the development of late postoperative renal dysfunction after postoperative day 30, defined as an increase in serum creatinine by at least 50%. Late kidney dysfunction was not thought to be plausibly related to the implantation of a particular IVBI, and was therefore considered an appropriate endpoint for use in Falsification Hypothesis testing.

For the propensity match analysis, a significant difference was considered present if the confidence intervals (CI) between two independent proportions, as measured by the Wilson method, did not cross zero16 when using an alpha of 0.05. For IPTW, adjusted odds ratio (ORadj) were considered significant for increased risk of adverse events if the ORadj was greater than 1.0, with a 95% CI excluding 1.0. All analyses were performed within the DELTA application, relying on R-based statistical packages.7–9

Results

Unadjusted outcomes

A total of 612 patients underwent thoracic or lumbar spinal fusion between April 1, 2015, and December 31, 2018, at a Lahey facility, of which 6 patients were excluded due to prior surgery on the thoracic or lumbar spine within the prior 12 months. The clinical characteristics of the remaining 606 patients identified for analysis are shown in online supplemental table S1 of the Supplemental content, along with the subgroups receiving CB, those receiving an alternative IVBI, and those receiving both CB and an alternative device. A total of 10 patients (1.65%) underwent reoperation within 1 year of the index surgery, 3 patients (0.5%) developed a surgical site infection, 105 patients (17.5%) were hospitalized within 1 year of their index surgery, and 3 patients (0.5%) died within 12 months of surgery (see online supplemental table S2 of the Supplemental content). In addition, 66 patients (10.9%) received blood transfusions immediately following surgery. The 158 patients who received both a CB as well as one of the alternative IVBI were excluded from subsequent safety analyses as outcomes could not be definitively attributed to a specific device type.

Prespecified propensity analyses

Patients treated with the CB IVBI, alternative IVBI, and those patients treated with neither during qualifying thoracic and lumbar spine fusion surgeries were identified from EHR derived data (see online supplemental figure S1 in the Supplemental content). Among the 136 patients receiving only CB during their qualifying surgery, 69 (51%) were successfully matched with patients in the comparator group (receiving an alternative IVBI) after performing a propensity score match. As shown in table 1, post-matching standardized differences were less than 0.10 for 13 of the 18 covariates, with 5 covariates having post-matching standardized differences ranging from 0.12 to 0.19, indicating suboptimal covariate balance.

Table 1

Covariate distributions prior to the propensity score match, after the match, as well as in the remaining unmatched CONCORDE Bullet treated patients

Covariate	Total study population			After propensity match			Unmatched exposures
	CONCORDE Bullet	Alternate IVBI		CONCORDE Bullet	Alternate IVBI		CONCORDE Bullet
	(N=136)	(N=98)	Std. diff	(N=69)	(N=69)	Std. diff	(N=67)	Std. diff
Female	52.94%	52.04%	0.018	50.72%	55.07%	0.087	55.22%	0.090
Age—median (IQR)	61.84±12.43	58.87±13.16	0.232	60.32±13.30	60.09±13.88	0.017	63.40±11.36	0.249
Body mass index—median (IQR)	30.03±5.69	29.29±5.80	0.130	29.86±5.52	29.75±6.13	0.017	30.21±5.89	0.062
Diabetes	13.97%	10.20%	0.116	14.49%	10.14%	0.133	13.43%	0.031
Heart failure	0.74%	1.02%	0.031	1.45%	1.45%	0.000	0.00%	–
Coronary artery disease	4.41%	4.08%	0.016	4.35%	4.35%	0.000	4.48%	0.006
Smoker (active)	13.24%	8.16%	0.165	8.70%	8.70%	0.000	17.91%	0.274
Preoperative opioid Rx	19.12%	27.55%	0.200	24.64%	18.84%	0.141	13.43%	0.288
ASA class 1–2	68.38%	72.45%	0.089	66.67%	75.36%	0.193	70.15%	0.075
ASA class 3–5	31.62%	27.55%	0.089	33.33%	24.64%	0.193	29.85%	0.075
Prior spine surgery	0.74%	0.00%	–	0.00%	0.00%	–	1.49%	–
Single-level	40.44%	44.90%	0.090	57.97%	55.07%	0.059	22.39%	0.779
Two-level	0.00%	1.02%	–	0.00%	0.00%	–	0.00%	–
Complex repair (>2 levels)	40.44%	6.12%	0.889	7.25%	8.70%	0.054	74.63%	1.881
No level identified	19.12%	47.96%	0.642	34.78%	36.23%	0.030	2.99%	0.889
Surgeon volume low	5.15%	27.55%	0.636	8.70%	11.59%	0.096	1.49%	0.332
Surgeon volume medium	72.79%	64.29%	0.184	75.36%	76.81%	0.034	70.15%	0.117
Surgeon volume high	22.06%	8.16%	0.396	15.94%	11.59%	0.126	28.36%	0.302

ASA class, American Society of Anesthesiology physical status classification; IVBI, intervertebral body implant; Std. diff, standard difference.

There were no significant differences observed between the clinical outcomes in the CB treated patients as compared with the patients treated with alternative intervertebral body device implantation in the propensity matched analysis, as demonstrated in table 2. Among the 69 patients treated with the CB in the matched cohort, only 1 patient required reoperation within 12 months, as compared with no patients among the matched control patients.

Table 2

Clinical outcomes of the propensity match analysis comparing patients treated with the CONCORDE Bullet with matched patients treated with alternative therapies

Outcomes: Propensity match analysis	CONCORDE Bullet (%)		Alternative IVBI (%)		Relative risk	95% CI	P value
	(n=69)		(n=69)
Primary endpoint:
Reoperation at 1 year	1	1.4	0	0.0	n/a	n/a	0.316
Secondary endpoints:
Mortality at 1 year	1	1.4	0	0.0	n/a	n/a	0.316
Any hospitalization at 1 year	15	21.7	7	10.1	2.14	(0.93 to 4.93)	0.063
Blood transfusion (postoperative)	7	10.1	3	4.3	2.33	(0.63 to 8.66)	0.189
Surgical site infection	0	0.0	0	0.0	n/a	n/a	n/a

IVBI, intervertebral body implant.

Online supplemental figure S2 in the Supplemental content provides the results of the prospective active surveillance analysis, demonstrating stability of the results by repeated analysis of the primary endpoint outcome, by calendar quarter, throughout the study period.

As shown in table 3, the OR for the risk of reoperation at 12 months was 1.01 (95% CI: 0.99 to 1.03) as estimated by IPTW metholody, indicating no significantly increased risk for CB relative to the prespecified alternative IVBI. An extended analysis of the primary endpoint of reoperation out to 2 years maintained these null findings (OR 0.998, 95% CI: 0.962 to 1.035, p value 0.91). In addition, there were no significantly increased risks for any of the secondary outcomes at 12 months.

Table 3

IPTW analysis—estimated ORs for the risk of adverse outcomes with treatment using the CONCORDE Bullet

Outcomes: IPTW analysis	OR	Lower CI	Upper CI	SE	P value
Primary endpoint:
Reoperation at 1 year	1.01	0.99	1.03	0.01	0.27
Secondary endpoints:
Mortality at 1 year	1.01	0.99	1.02	0.01	0.40
Any hospitalization at 1 year	1.07	0.98	1.16	0.04	0.11
Blood transfusion (postoperative)	0.92	0.83	1.01	0.05	0.08
Surgical site infection	n/a	n/a	n/a	n/a	n/a

IPTW, inverse probability of treatment weighting.

Additional sensitivity analyses

The eligible patient population was expanded in order to perform several post-hoc sensitivity analyses. Patients without a specific comparator device of interest, including patients with no device implanted at the time of index surgery (identified by procedure codes in online supplemental tables S7 and S8), as well as those patients undergoing complex repair (covering more than two spine levels) at the time of their index surgery were included in the analyses. Propensity score matching and IPTW analyses were performed on these additional groups (see online supplemental tables S3-S5 in Supplemental content) and the results were consistent with the primary analysis, with no increased risk of adverse events associated with the use of the CB.

Finally, the results of the Falsification Hypothesis comparing CB and alternative IVBI patients demonstrate a relative risk of 0.67 (95% CI: 0.11 to 3.87, p value 0.017) of developing renal dysfunction in the year following index surgery, indicating no evidence of significant residual confounding in the cohorts analyzed. Residual confounding was also absent in the post-hoc sensitivity analysis given the observed relative risk of developing renal dysfunction (0.25 with 95% CI: 0.15 to 2.75, p value 0.472) for the expanded pool of patients in the CB group compared with patients not exposed to CB.

Discussion

The CB DELTA study was designed to assess the feasibility of applying active surveillance tools to accumulating clinical data derived from the EHR to evaluate the safety of a commonly used IVBI. We compared the safety profile of different IVBI devices, through the application of rigorous, validated, prospective, active surveillance methods tools using clinical data collected during the routine course of care. During the study period of 34 months, 136 patients underwent thoracic or lumbar spine surgery with the implantation of the device of interest, while 212 patients were treated exclusively with alternative spinal stabilization devices. Overall, our analysis showed similar 12-month rates of reoperation, mortality and any cause rehospitalization in the CB and alternative implant groups. The findings were consistent among subgroups analyzed and were confirmed through IPTW analysis, in addition to propensity score matching.

The findings of this study are significant for two reasons. First, this NESTcc feasibility study further demonstrates the utility of integrating routinely collected EHR data into prospective, active safety surveillance activities for medical devices. Using well-structured clinical data available in our hospital system, we were able to validate the device exposure, key clinical covariates and relevant outcomes, suggesting that safety surveillance of spinal implants is feasible using routinely collected EHR data. Second, there is limited comparative safety data of spinal surgery techniques, and this analysis is among the first reports comparing the safety of a commonly used IVBI with that of risk-adjusted comparator treatments. As the number of options for interbody devices expands, the ability to generate clinical effectiveness and safety data regarding fusion rates and reoperation rates from EHR data will allow providers to better assess the comparative safety, effectiveness, and costs of different treatment strategies.

Limitations

There are several important limitations of this study. The patients studied were treated within a single healthcare system, by a small group of experienced spine surgeons, limiting the generalizability to different environments. Also, the sample size of patients treated with the CB and alternative spine implants were limited, thereby reducing the power of the analysis to identify small differences in adverse outcome rates. However, the application of IPTW methods to reduce the loss of information by including all treated patients, suggests that the sample size did not materially impact the findings. In addition, additional patient groups were explored as comparators, with similar findings, further supporting the primary results. In addition, the residual imbalance in clinical risk factor distribution noted between the treated and comparator groups limit the accuracy of the estimated differences in adverse event rates.

Additional limitations include the risk of residual confounding, which we sought to minimize through robust risk adjustment using propensity matching and propensity weighting. In support of the effectiveness of the methods, the prespecified Falsification Hypothesis analyses found that late renal dysfunction, thought unrelated to choice of surgery, occurred with similar frequency between the matched populations, thereby indicating that there was no evidence for significant residual confounding in the cohorts analyzed.

This study was supported through a competitive research grant from NESTcc in accordance with Federal research regulations, and the study steering committee included regulatory and quality representatives from Johnson & Johnson, the manufacturer of the device studied.