Determining value of Coordinated Registry Networks (CRNs): a case of transcatheter valve therapies.

BACKGROUND: The Transcatheter Valve Therapy (TVT) Coordinated Registry Network (CRN) supported 23 regulatory decisions and ensured evidence-based evaluation of the application of TVT technology. However, there are cost concerns that require value assessment of the TVT CRN compared with traditional study designs.
OBJECTIVES: We aimed to determine the value created by the TVT CRN based on (1) Return on investment (ROI), (2) Time saved (TS) in conducting necessary regulatory studies.
METHODS: For both ROI and TS analyses, we compared studies that used the TVT CRN with those that would have been required if the registry did not exist (counterfactual studies). To estimate ROI, we accounted for the costs of investment and gain from investment. Both the counterfactual costs and length of studies were projected using design specifications determined by US Food and Drug Administration (FDA) reviewers.
RESULTS: We identified 21 studies using the TVT CRN (supporting 23 FDA decisions) that generated evidence on TVT for three device manufacturers. ROI is estimated to be greater than 550%. TS by using the CRN ranged from months to years.
CONCLUSIONS: The CRN method to evidence generation creates value for manufacturers and the broader device ecosystem, demonstrated with this example of the TVT CRN. The public health benefits of evidence created by this CRN outweighs the difference in data quality between traditional clinical studies and the CRN method. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Access to reliable and meaningful evidence about the safety, effectiveness, and quality of medical devices is essential to inform care and improve patient outcomes—a goal shared by stakeholders in the medical device ecosystem including patients, clinicians, health systems, payers, device manufacturers, and regulators. Traditional methods of evidence generation for device evaluation are very costly, time-consuming and have well-understood limitations.1 2

The rapidly developing digital health information infrastructure in the country is increasingly being harnessed to support more reliable, affordable, and timely evidence generation. The Coordinated Registry Networks (CRNs), as proposed by the National Registry Taskforce, strategically bring together real-world data from a variety of sources to support improved device evaluation.3

This paper describes the use of one CRN, the Transcatheter Valve Therapy CRN (TVT CRN), which has been successfully used for regulatory decision-making during the past several years. The TVT CRN links a cohort of persons who received TVT enrolled in the national Society for Thoracic Surgery/American College of Cardiology (STS/ACC) TVT Registry with Centers for Medicare and Medicaid Services (CMS) claims data (linked registry studies). The STS/ACC TVT Registry was created through the National Cardiovascular Data Registry (NCDR) via a partnership of the STS and the ACC, in close collaboration with the Food and Drug Administration (FDA), the CMS, and the Duke Clinical Research Institute.4 As of March 2019, the TVT CRN includes more than 600 sites that provide TVT, reporting to the STS/ACC TVT Registry, with over 195 000 individual patient data entries.5

The objective of this study was to demonstrate value created by CRNs in three areas: public health benefits through improved evidence, return on the investment (ROI) for conducting studies, and ‘time saved’ (TS) in conducting studies. The TVT CRN studies were conducted in lieu of traditional studies (a mixture of preapproval studies and postapproval studies (PASs)). We compared use of the TVT CRN (observed studies) to studies that would have been conducted had the TVT CRN not existed (counterfactual studies).

Methods

Data sources and approach

We identified and collected information on studies in the TVT CRN submitted to the US FDA between 1 December 2011 through 30 September 2017 and led to a regulatory decision.6 7 Regulatory decision was defined as approval of premarket approval application (PMA) supplements or postmarket requirements as conditions of approval. Table 1 lists the counterfactual studies that likely would have been conducted if the TVT CRN did not exist. Counterfactual studies included investigational device exemption (IDE) studies for expanded indication, extended postmarket follow-up of IDE studies (traditional methods), and traditional new enrollment PASs, those replaced by studies using the TVT CRN. Not all the studies were included in the model for ROI as some were combined for efficiency in fielding studies. In practice, studies are frequently combined when fielding; for calculation of ROI, therefore, we excluded 6 of the 22 potential counterfactual studies providing 16 studies (A through P listed in table 1) to be included in the ROI calculation.

Table 1

Costs and drivers of counterfactual studies: The TVT CRN*

Counterfactual studies	Total subjects	Total sites	Study duration (years)	Evaluations per year	Total procedure count	Total cost per study
A	2000	39	5	1	5	$ 21 949 424
B	550	27	5	1	5	$ 7 539 932
C	1700	27	5	1	5	$ 18 814 232
D	1000	28	5	1	5	$ 11 911 048
E	200	28	5	1	5	$ 4 165 448
F	1000	57	10	1	7	$ 17 798 692
G	1000	29	5	1	5	$ 11 925 264
H	1000	57	10	1	7	$ 17 798 692
I	1000	45	5	1	5	$ 12 152 720
J	200	37	5	1	5	$ 4 293 392
K	100	41	5	1	5	$ 3 382 056
L	1000	65	10	1	7	$ 17 979 140
M	150	29	5	1	5	$ 3 695 564
N	150	29	5	1	5	$ 3 695 564
O	150	29	5	1	5	$ 3 695 564
P	100	29	5	1	5	$ 3 211 464

*Note. The calculations assume all patients are treated inside USA, requiring inperson evaluations, and the study was not randomized and had no control group. The letters designating the individual studies correspond with the studies listed in table 3.

Table 3

Counterfactual study* times, real scenario study times, and days (years) saved

Study	Study type	Total counterfactual study time (days)	Observed days specified by design	Days saved	Years saved
A	PAS	5569	1825	3744	10.3
B	PAS	3312	1825	1487	4.1
C	PAS	6421	1825	4596	12.6
D	PAS	4432	1825	2607	7.1
E	PAS for a labeling expansion	2346	1825	521	1.4
F	PAS for a labeling expansion	4931	3650	1281	3.5
G	PAS	4342	1825	2517	6.9
H	PAS for a labeling expansion	4931	3650	1281	3.5
I	PAS for a labeling expansion	3447	1825	1622	4.4
J	PAS for a labelling expansion	2220	1825	395	1.1
K	PAS for larger size value	2003	1825	178	0.5
L	PAS for a labeling expansion	4773	3650	1123	3.1
M	PMA study for labeling expansion	2203	1825	378	1.0
N	PMA study for labeling expansion	2203	1825	378	1.0
O	PAS for labeling expansion	2203	1825	378	1.0
P	PAS for labeling expansion	2077	1825	252	0.7
Q†	PAS for minor iteration	2585	1825	760	2.1
R†	PAS for precaution removal	2585	1825	760	2.1
S†	PAS for minor iteration	3084	1825	1259	3.4
T†	PAS	3605	1825	1780	4.9
U†	PMA study for labeling expansion bundled	635	365	270	0.7
V†	PAS for labeling expansion	635	365	270	0.7

*Note. The counterfactual studies listed here include all possible counterfactual studies needed to support the regulatory decisions discussed in this paper. Table 1 represents a subset of these studies, which were used in the calculation of ROI.

†Studies not included in the ROI calculation.

PAS, postapproval study; ROI, return on investment.

Costs of counterfactual studies

We calculated an estimated cost of each counterfactual study using a model described by Wimmer et al.8 Cost drivers in the Wimmer et al. model include: study size, number of sites, need for recruitment, need for randomization, need for control groups, total number of sites outside USA (as a percentage), study duration (years), number of patient evaluations per year, patient evaluation type (phone/inperson), follow-up procedure required (yes/no), total procedure count, type of procedure (if yes), and organ system. We reduced the duration of the simplified base case to 1 year (Dr Fredrick Resnic personal communication). The premarket and postmarket review staff at FDA (expert opinion) provided the design features of counterfactual studies (what premarket and postmarket studies would have looked like if the TVT CRN had not been used) for calculation of the costs using the Wimmer et al. model.

ROI and TS

ROI. We defined the ROI as the cost of savings from investing in the CRN divided by the cost of investment in the CRN, multiplied by 100. The cost savings is the sum of costs of the counterfactual studies subtracted from the costs of observed studies conducted in the CRN.

TS. The time interval used for calculating TS is the number of days from enrollment of the first patient in the study to the date that follow-up is complete on the last study participant, the clinical data generation period in figure 1.

Figure 1

Time frame from study design to marketing application submission. IDE, investigational device exemption; IRB, Institutional Review Board

The focus on the clinical data generation period frames a metric that can be meaningfully compared across CRNs, or between CRNs and counterfactual studies, because it provides for standardization. Other intervals, which are variable between studies were not studied.

We calculated TS by using the CRN approach by subtracting the days estimated for each counterfactual study from the number of days required for clinical data generation in the observed studies. TS was calculated on each study that might have been conducted, irrespective of combinations of counterfactual studies used for calculation of ROI.

The number of days required for the observed study is the number of days specified in the study design. For counterfactual studies, the estimate of days required to complete a study is the sum of the days of follow-up specified by the design and the days required for enrollment. The counterfactual study enrollment time (CET) was calculated by applying the rate of enrollment of the pivotal study (a traditional premarket study) to the specifications of the counterfactual study (study size and specific length of study called for by the design) using the following formula:

CET = (number of patients for study by design / rate of enrollment) / study duration specified by design

The rate of enrollment in traditional studies was derived from the Summary of Study and Effectiveness Data description of study design. Specifically, two TVT premarket studies were compared which determine the patients per site per year (both studies enrolled five patients/site/year). This estimate was used for the calculation of counterfactual enrollment times.

Results

The TVT CRN supported 21 studies, which led to 23 decisions by the FDA; data for two submissions were derived from a common study. The TVT CRN was used to support four premarket decisions, including expanded indications, and 19 postmarket surveillance studies in lieu of traditional PASs (table 1). The cost to the three medical device companies for these studies conducted in the TVT CRN is a total of $25.05 million paid to NCDR. This expense of $25.05 million is the cost of the observed the TVT CRN-based regulatory studies.

The costs and some of the study design features for each of the 16 counterfactual studies included in the ROI calculation are provided in table 1. These counterfactual (traditional) study designs included a planned enrollment ranging from 100 to 2000 patients, recruited from 27 to 65 participating US sites, who were to be followed for a median of 5 years. The estimated costs of the counterfactual studies ranged from $2.16 million to $18.93 million with a median estimated cost of $8 million (IQR: $2.59–$12.55 million) and a total projected cost of $164 million.

Return on investment

The cost savings associated with performance of the observed studies using the TVT CRN, rather than the counterfactual (traditional) study design, is estimated to be $138 958 196 ($164 008 196 for the counterfactual studies less the $25.05 million spent by the medical device manufacturers to support the TVT CRN studies performed). Thus, the ROI for the TVT CRN studies is 555% (see figure 2).

Figure 2

Calculating return on investment (ROI) for Transcatheter Valve Therapy (TVT) Coordinated Registry Network (CRN) studies as compared with the counterfactual (traditional) study design.

A sensitivity analysis was performed to demonstrate the robustness of the model used to calculate ROI that increased and decreased (by 5% and 10% in each direction) the numbers of study subjects and the numbers of study sites specified by the counterfactual studies (table 2). Varying the number of subjects creates a range of ROI from 603% to 507%. Note that the original calculations were based on assumptions about meaningful effect sizes, so that reducing sample size beyond a certain point would clearly lead to underpowered studies, which would not typically be accepted by the FDA. Varying the number of sites creates a range of ROIs from 559% to 551%. The table also includes ROI estimations that vary the number of counterfactual studies included in the estimate. A high estimate (682%) includes all possible counterfactual studies. The low estimate (368%) assumes studies would have been conducted (combining individual studies that would have been needed, based on field efficiencies). The lower estimate assumes combining a higher number of studies than is probable.

Table 2

Return on investment (ROI) sensitivity analysis: variation of number of subjects, sites, and counterfactual studies used to calculate ROI

Parameters	Variations	Compared with ROI 555%
Subjects	Plus 10%	603
	Plus 5%	579
	Minus 5%	531
	Minus 10%	507
Study sites	Plus 10%	559
	Plus 5%	557
	Minus 5%	553
	Minus 10%	551
Counterfactual cases	>counterfactual studies 22, maximum, no combined studies	682
	<counterfactual studies than 10, combining 12 studies	368

Time saved

Table 3 presents the time (days and years) required to complete observed studies conducted with the TVT CRN as compared with the days required to conduct the counterfactual cases. TS for studies varied between 0.5 years and 12.6 years (median=2.6, IQR: 1.0 years to 4.4 years). The PMA studies for labelling expansion (M, N, and U) saved 1.0 year, 1.0 year, and 0.7 years, respectively. The slow rate of enrollment in traditional (counterfactual) studies compared with the TVT CRN is the major factor driving these calculations.

Discussion

The ROI and TS by using a CRN in place of traditional study designs for evidence generation for TVT was estimated to be 555% and was robust to changes in assumptions. The lowest estimate of ROI (368%) came from reducing the number of studies that were assumed to be required to address the relevant regulatory questions. As noted in more detail below, the ROI calculated for the TVT CRN cannot be used to generalize value created by other CRNs. While the methods used here are presented as robust, comparisons of case studies should be conducted to understand the broader utility of the CRN approach to evidence generation. There will likely be variability in the degree to which CRNs are cost-saving.

Underestimating ROI

The model and assumptions used to calculate ROI may be an underestimate for the following reasons:

The Wimmer et al. model was created to estimate the cost of PASs. It is broadly understood that premarket studies are much more expensive than PASs, due to several study specifications that are more stringent for premarket studies, thus suggesting the ROI is an underestimate.9

Dropping some of the counterfactual cases from the ROI calculation is justified by the reality that studies may be combined when fielding studies. The site enrollment costs decrease when studies are combined. For ease in calculation, the dropped studies from the ROI calculation include dropping both the cost of site enrollment and the cost of the required subjects. The cost of subjects would be borne in combined traditional studies but is not accounted for in the calculation, thus producing an underestimate of the ROI.

The national coverage decision by CMS to pay for TVT was also based on findings from the TVT CRN. The benefit to companies was not included in the calculation in this paper and further contributes to underestimation of the ROI.

Other considerations

The sources of value created by the CRN are derived by leveraging the substantial investment made by hospitals in the registries and by the linkage of outcome data between the registry and CMS (Medicare claims). Outcome data over time are very expensive to obtain using traditional methods and were provided at no cost to the CRN through the linkage of CMS claims records.

This study did not evaluate value created by registry for hospitals; those investments are made for reasons including: quality improvement, comparative effectiveness research, and provision of evidence to negotiate with payers. A study by Ahmad et al. documents ROI for registries at Dartmouth hospital.10 The ROI for hospitals and clinical centers maintaining registries will increase as data collection and curation in clinical settings become more efficient (less costly). Automation of registry work (structured data capture and natural language processing, approaches) is being actively pursued by many researchers.11

STS/ACC TVT Registry data quality is enhanced and promoted through NCDR’s use of aggressive registry site education, data quality reports, data audit strategies along with adjudication of key adverse outcomes. Data in the STS/ACC TVT Registry have been validated.12 The CMS claims data used in the TVT CRN for evaluating longitudinal outcomes have also been validated.13–16 The CRN approach and traditional studies are not considered to be identical or equivalent. The quality, variety, and richness of data collection from traditional studies are not available through the CRN, though, on balance, the judgment of FDA review staff is that the overall improvement in the strength of evidence derived from the TVT CRN outweighs the limitations of the data.

Study sizes that support an understanding of heterogeneity of patients and operating context must be large enough to capture heterogeneity (e.g., across sites) and support analysis to evaluate device benefit and risk. That said, the mechanism for case selection into a registry is crucial. The important aspect is that selection into the registry should ensure that the included patients are representative of the underlying population of interest, and is not, for example, determined by including only those patients whose prognosis is more favorable than that of the underlying population. Entering consecutive patients into the registry, in an unselected manner, from a broad range of sites, is often a preferred way to achieve this representativeness. In the case of the TVT CRN, nearly all cases are included in the registry.

The large number of sites in the TVT CRN saved substantial time in enrollment compared with the traditional studies. The specified length of each counterfactual study is determined by design, but subject enrollment times are variable, which is the source of ‘time saved’ in these calculations.

The TS measured here may be an underestimate for two reasons. First, the metric does not include a measure of time to enrol sites, as those data are not available for this study. Clinical trialists understand that the time needed to enlist sites varies greatly among studies. Documentation from a study of enrollment time in two PASs of a drug-eluting stent (one using registries compared with a second using traditional methods) shows similar dramatic time savings.17 The time required for initial enrollment of sites into the registry must be considered. However, if the registry is already in existence, then subsequent use of the operating registry saves time. If the registry must be built from scratch, then the time saving increases after initial establishment of the registry.

The second reason we believe the TS is an underestimate is that, in the case of PASs, enrollment rates would be expected to be lower than those for PMA studies. It is more difficult to enrol a patient into a study when the device is already commercially available. In the calculations presented, we assume enrollment rates of PMA studies for all the counterfactual studies.

The value (in dollars) derived from TS differs for premarket and postmarket studies. For label expansion studies, the TS contributes to additional ROI, as faster time to market may translate into increased revenue. The revenue generated by faster time to market is different for each device because of the size of the market that becomes available with label expansion, the degree of market penetration, the margin on the specific device, and the reimbursement status. Business decisions also determine how and when this evidence is used. For these reasons, TS in the label expansion studies do not automatically translate into additional ROI for companies. TS in a PAS should be considered as an efficiency but does not directly contribute to ROI.

The label expansions were not planned as part of the original effort to use the TVT CRN for PASs, raising some issues for consideration. Importantly, collecting patient data through a national registry such as the STS/ACC TVT Registryof device off-label use represents an important advance in the evaluation of safety and effectiveness of patient populations previously not consistently or thoroughly assessed. Caution is needed, though, to avoid encouraging inappropriate off-label use, solely for the sake of data generation, as this may raise ethical concerns as well as constitute an investigation subject to regulation.

The meaning of the TS metric may be appreciated by comparing the time saving of the CRN approach to efforts to save time in FDA reviews. Major efforts in monitoring and decreasing review times are well documented.18 Increasing the time efficiency of the review process has resulted in days saved.

The dollar amounts for savings for the TVT CRN case study cannot be directly extrapolated to other device areas. Absolute costs of studies vary widely by device. The ROI as a relative rate comparing registry versus clinical studies provides a more useful comparison.

Public health benefit

The public health benefits of using CRNs stem from robust generalizable study populations and sites, more timely results, and a move towards real-time evidence device generation. The large study sizes made possible by the TVT CRN provide more representative data of a full range of patients and clinical settings, compared with traditional studies. Additionally, data coming from the large number of sites provide an understanding of learning curve effects, diversity in operator effects including volume, and variations in outcome created by differences among implant techniques, concomitant therapy, or hospital settings.

On-going data colleciton (both new patients and outcomes) in the TVT CRN creates near real-time surveillance of the population exposed to the device.19 This active surveillance system can detect problems with devices sooner than traditional approaches currently used (passive surveillance). This improved comprehensive surveillance, the faster detection of device problems, is a public health benefit to patients, clinicians, and industry.

There are also clear benefits to patients who received TVT as a life-saving procedure based on the label expansions made possible with the TVT CRN. To quantify this public health benefit, an analysis can be envisioned that projects the number of person-years of lives extended (e.g., Disability Adjusted Life Years DALYs or Quality Adjusted Life Years QALYs) due to the new device in the new indication.