Creation of objective performance criteria among medical devices.

Objectives: Objective performance criteria (OPC) may serve as a tool to expedite the approval process and continue active surveillance of class III medical devices. Thus far, published guidance on the creation of OPC has been clinical area-specific. This study aimed to capture reflections from key stakeholders on the creation of OPC that may serve as a precursor for a formalized conceptual framework within the USA. Design: Reflections from key stakeholders and guidance from an advisory committee were captured to gain an understanding of the elements that are crucial to the generation of OPC. Setting: A non-probability sampling method using the purposive sampling strategy was employed to identify relevant stakeholders for engagement in semi-structured, open-ended, concept elicitation discussions. Participants: Stakeholders involved in the generation of OPC. Main outcome measures: Elements and themes regarding the priorities of, experiences with, roles within and perceived challenges associated with OPC creation captured through a phenomenological approach.
Results: A total of 27 participants were engaged to represent the following contributors: representatives of registries, health systems, health technology assessment bodies, clinicians, device application reviewers, payers, patients, patient representatives, patient caregivers, device manufacturers, data coordinators, data analysts and data informaticians. Consensus was achieved on the five core elements: (1) identification of medical devices, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings. The engagement of key stakeholders (38%) was cited most frequently as the most important core element. Access to meaningful and high-quality data sources (47%) was the most frequently mentioned challenge. Conclusions: The reflections from the participants identified five elements to be considered when generating an OPC within class III medical devices and may provide the needed foundation for the development of official guidance on OPC generation. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Objective performance criteria (OPC) may be leveraged to expedite the approval process and continue active surveillance. Currently, there are no frameworks detailing the necessary steps for the creation of OPC within class III medical devices.

Stakeholder engagement identified five elements that should be considered when generating an OPC: (1) identification of medical devices, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings.

The identified elements may provide the needed foundation to organizations developing clinical area-specific guidance, to government agencies providing guidance related to OPC for regulatory purposes, and to investigators contributing to OPC generation.

Introduction

The Food and Drug Administration (FDA) is both one of the most vested stakeholders and the primary regulator for medical devices in the USA. The 21st Century Cures Act and the Food and Drug Administration Reauthorization Act require the FDA to decrease the approval time of devices while continuously ensuring that all approved devices are safe and effective.1 2 This expectation is especially daunting when considering class III medical devices given that these devices are implanted in a patient for several decades, and their primary purpose is to sustain life or prevent significant disability. New regulatory approaches, such as the creation of objective performance criteria (OPC), which allow for the swift approval of new devices while simultaneously ensuring that all devices entering and currently on the market demonstrate a reasonable assurance of safety and effectiveness have been explored.3–6 These numerical targets of outcomes of interest are derived from existing data and can be leveraged as comparators for the assessment of safety and efficacy.6

Many stakeholders benefit from OPC during the premarket approval process and postmarket evaluation. Given that medical devices evolve incrementally, approved medical devices currently on the market may serve as suitable comparators to devices seeking approval. OPC derived from devices present on the market can be used as target measures in single-arm clinical trials.6 7 Single-arm trials may be favorable to some stakeholders because they avoid randomization and allow all patients in a clinical trial to receive the novel device. In addition, device manufacturers may benefit from shorter and less costly trials, while clinicians may receive access to novel devices for their patients faster.5 Following a device’s approval, OPC are one way for regulatory bodies, the clinical community, reimbursement agencies and patients to monitor how devices perform outside of a trial setting through benchmarks and provide target objective performance measures for new devices. This may aid regulatory bodies in identifying devices that may need additional postmarket action including safety communication, mandated postmarket studies or compliance action. Patients, clinicians, health systems and payers may use OPC to inform treatment-related and reimbursement-related decisions. Registry representatives, data analysts and informaticians can leverage and strengthen their data sources to accurately inform OPC, potentially expanding the use and strengthening the impact of their data sources.

There are currently no specific documents or publications within the USA) exploring stakeholder perspectives regarding the needed considerations for the creation of OPC within the medical device space.4 8 The purpose of this study is, therefore, to capture reflections from key stakeholders on the creation of OPC that will serve as a precursor for a formalized conceptual framework for the creation of OPC within the USA with general criteria that may be applied to class III medical devices.

Methods

Advisory committee

A multistakeholder advisory committee was established and met regularly to guide and oversee all aspects of this work. The advisory committee included representatives of academia, government, regulatory bodies, practising clinicians and a representative of a medical device registry used for the development of OPC and regulatory decision making.

Stakeholder engagement

Key stakeholders were engaged to gain an understanding of the elements they believed are integral to a framework. Key stakeholders included registry maintenance representatives, health system representatives, clinicians, device application reviewers, payers, health technology assessment (HTA) body representatives, patients, patient representatives, patient caregivers, device manufacturers, data coordinators, data analysts and data informaticians. Payers and HTA bodies were combined into one stakeholder group since many stakeholders were previously employed by an HTA body or performed value assessment within the payer organisation.

Participants and procedure

A non-probability sampling method using the purposive sampling strategy was employed to identify relevant stakeholders for engagement in semistructured, open-ended, concept elicitation discussions.9–11 Identified relevant stakeholders were invited to participate in discussion through emailed invitations. Concept elicitation discussions were set to a minimum of 25 stakeholders. If necessary, further stakeholders were identified using chain sampling.12 Stakeholders that agreed to participate were asked to identify further stakeholders that would be willing to participate. Stakeholders were engaged until no new concepts were uncovered and saturation of framework elements was reached.10 The stakeholders were asked six questions pertaining to the types of elements that need to be present in the framework, the importance of the elements, feedback on the proposed elements of the framework, the stakeholders crucial to the creation of OPC, the challenges associated with the creation of OPC and any additional information critical to a comprehensive framework. The questions were developed with the guidance of the advisory board and are outlined in table 1. The initial proposed elements of the framework were informed by clinical area-specific published guidance on OPC creation, generation, application and discussion of OPC as well as the input from the advisory board.3–5 13–15 Discussions with each individual stakeholder were held either in person or over the phone in a private setting and lasted approximately 60 min. Notes were taken during discussions with the key stakeholders. A read-back method was used to ensure that all concepts mentioned by the stakeholders were accurately captured. The discussions were not audiorecorded to allow stakeholders to engage in honest informal discussions without fear of professional repercussions.

Table 1

List of questions for key stakeholders

	Questions for key stakeholders
1.	What should the elements of the framework be? Considering the importance of these elements, how would you rank them?
2.	Comments on current elements of the framework: Identification of medical devices for the development of objective performance criteria (OPC), engagement of key stakeholders, selection of data source, performance of appropriate statistical analyses, reporting of findings.
3.	From the stakeholder’s perspective, which element of the framework is most crucial?
4.	Which stakeholders are crucial to the creation of OPC?
5.	From the stakeholder’s perspective, what is the biggest challenge in the creation of the OPC?
6.	Is there anything that is essential to discuss or take into account in the creation of the framework?

Coding process and analyses

A phenomenological approach was implemented to summarize and describe stakeholder perspectives regarding the priorities of, experiences with, roles within and perceived challenges associated with OPC creation.16 Discussion notes were thematically analyzed using short phrases and entered into a codebook.17 The codebook was modified until no new codes were uncovered.17 A saturation grid was employed to ensure that saturation was achieved between stakeholders within the framework elements.10 Following all discussions with key stakeholders, the initial proposed elements were modified to reflect additional input from the key stakeholders (figure 1).

Figure 1

Methodology for the capture of stakeholder reflection regarding the creation of objective performance criteria with general criteria to be applied to class III medical devices.

Results

Thirty individuals within the stakeholder groups were approached. Eight individuals declined; therefore, five further individuals were identified using chain sampling. Of the eight individuals who declined, five represented regulatory bodies and three represented payers. In total, 27 stakeholders participated in this study. Six participants represented more than one stakeholder type. One participant represented three stakeholder types. The number of participants representing each stakeholder is summarized in table 2.

Table 2

Summary of stakeholders, stakeholder types, roles in and effect of objective performance criteria (OPC) generation

Stakeholder types	Role and effect of OPC generation	Number
Engaged stakeholders
Registry representative	Registries collect the real-world data needed to generate OPC. Registry representatives can leverage the OPC to generate feedback reports to clinicians and patients on the performance of devices to inform future decision making.	2
Health system representative	Health system representatives collect data on the use and performance of devices within their health system. This data can be used to create OPC and inform clinicians within the health system on the performance of devices to inform future clinical decision making.	3
Clinician	Clinicians aid in the generation of real-world data needed to appropriately evaluate the medical devices. Clinicians may disseminate the findings of the OPC to their patients in order to inform joint clinician and patient decision making.	3
Patient and patient caregiver	Patients and patient caregivers provide input on meaningful endpoints needed for OPC creation. Patients are the primary users of devices and are most affected by devices brought onto the market using OPC. Furthermore, OPC may be used to aid in decision making with regard to treatment.	5
Device manufacturer	Device manufactures are the primary suppliers of devices. The manufacturers use the created OPCs as a comparison when seeking approval for devices seeking approval. The OPC may be useful in terms of post-market studies required by regulatory bodies.	3
Device regulatory application decision-maker	Regulatory decision-makers may use the created OPCs in their evaluation of a marketing application and can leverage OPC to identify devices that may be considered for removal from the market following approval.	3
Data coordinator	Data coordinators link and house the real-world data needed for OPC generation. Data coordinators identify data sources that can be leveraged to comprehensively evaluate medical devices and identified meaningful endpoints.	3
Data analyst	Data analysts generate real-world evidence from real-world data. They employ appropriate statistical methods to analyse data and create robust OPC.	3
Data informatician	Data informaticians aid in the collection of data elements needed for the generation of OPC. They aid in establishing the infrastructure for the linkage and harmonization of the needed data sources to generate OPC for clinically meaningful endpoints.	2
Payer/health technology assessment body	Payers and health technology bodies use OPC for reimbursement decision-making and value assessment. These decisions may affect which medical devices clinician and patient access to medical devices.	7
Advisory committee
Academia	Conduct studies using robust methods for the creation of OPC	3
Government -regulatory body	Provide input to the development of and use the created OPCs (1) to aid in the determination of safety and efficacy of a device seeking approval and enhancement of postmarket surveillance; (2) to augment the tools available to registries and CRNs; and (3) to promote the application of the tool in other clinical areas.	1
Registry representative	Provide data used to create OPC	2
Clinician	Use OPC for clinical decision making	2

Consensus was achieved among all participants on five elements (figure 2). Within the first core element, Identification of Medical Devices for the Development of OPC, participants emphasized the need for device selection to be a conscientious and careful process as well as the need for the device to be mature enough with sufficient real-world data. During the discussion of the second core element, Identification and Engagement of Stakeholders, seven out of the 10 stakeholder groups stated that OPC generation is better accomplished as a collaborative effort. Regulatory bodies, device manufactures and clinicians were the most frequently mentioned stakeholders needed to be involved in OPC generation. Regarding the selection of an appropriate data source, 70% of participants stated that they prioriticed data sources that capture clinically meaningful and relevant outcomes. In the fourth data element, performance of appropriate statistical analyses, the determination and definition of appropriate endpoints encompassing safety and effectiveness was discussed by 80% of the stakeholder groups. Finally, 70% of stakeholder groups stated that a discussion regarding the operationalization of the OPC was needed when reporting findings. The need for transparent reporting of all analyses and findings was mentioned by 60% of the stakeholder groups. The saturation grid summarizing elicited concepts and demonstrating saturation between stakeholder groups within the framework elements is presented in table 3. Each core element consists of multiple subelements outlined in table 3.

Figure 2

Identified core elements of objective performance criteria (OPC) development.

Table 3

Saturation grid of elements identified through published literature and engaged stakeholders (pages 1–2 of 3)

	Previously published literature	Registry representative	Health system representative	Clinician	Patient and Patient caregiver	Device manufacturer	Device application reviewers	Data coordinator	Data analyst	Data informatician	Payer/HTA	Total
Identification of medical devices for the development of OPC	X	X	X	X	X	X	X	X	X	X	X	11
Device selection is a careful and conscientious process			X	X				X	X	X		6
Sufficiently mature device with sufficient collected real-world data	X						X	X	X		X	5
Sufficient level of understanding associated with the technology							X	X	X			3
Natural history of the indication understood						X	X		X			3
Priority given to medical devices with specific characteristics	X							X	X			3
Previously published literature ought to be reviewed for existing OPC	X	X	X			X						4
Engagement of key stakeholders	X	X	X	X	X	X	X	X	X	X	X	11
OPC generation is a collaborative effort	X	X	X	X		X	X	X	X			8
Stakeholders involved												–
Regulatory and notified bodies		X	X	X	X	X	X	X	X	X	X	10
Industry/device manufacturers		X	X	X	X	X	X	X	X	X		9
Clinicians		X	X	X	X	X	X	X		X	X	9
Patients				X	X	X	X			X	X	6
Data owners		X	X									2
Payers and HTA bodies		X		X		X	X	X			X	6
Hospital health systems								X			X	2
Professional organizations					X				X		X	3
Epidemiologists and analysts				X					X	X		3
Data informatician										X		1
Multistakeholder collaborative a priori decision making		X	X	X		X	X	X	X			7
Determination of minimally clinically important differences		X	X					X	X			4
Selection of an appropriate data source	X	X	X	X	X	X	X	X	X	X		10
Differing data sources for OPG (Objective Performance Goals) versus OPC (Objective Performance Crtieria) creation	X						X	X	X			4
Prioritize data sources with standardized data elements and libraries		X	X					X		X	X	5
Prioritize data sources that capture clinically meaningful relevant outcomes to patients	X		X	X	X	X		X	X		X	8
Data quality assessed using the IMDRF’s (International Medical Device Regulators Forum) eight characteristics of a registry	X									X		2
Consider national registries, international registries, claims and linked data sources	X		X	X		X						4
Performance of appropriate statistical analyses	X	X	X	X	X	X	X	X	X	X	X	11
Identification of the study Population												–
Clearly define the study cohort with appropriate inclusion and exclusion criteria	X			X			X			X		4
Consult stakeholders, expert opinion and literature to determine appropriate inclusion/exclusion criteria	X			X						X		3
Required sample size needs to be statistically justified and hypothesis driven	X											1
Endpoints												–
Assess and include effectiveness and safety endpoints	X	X	X	X	X	X	X			X	X	9
Discuss the determination of appropriate endpoints and their definitions	X	X	X	X	X	X		X		X	X	9
Include short-term and long-term outcomes	X	X		X	X	X				X	X	7
Select endpoints relevant to the patient	X	X		X	X	X	X			X	X	8
Engage patients to capture any endpoints due to unintended consequences of the device			X	X	X	X				X		5
When available, include functional outcomes such as patient-reported outcomes	X			X		X	X			X		5
When possible, assess soft endpoints		X		X								2
When possible, assess quality endpoints				X						X		2
Assess endpoints at relevant time points to provide suitable comparisons in single-armed trials	X				X						X	3
Assess long-term endpoints at the most prolonged time possible	X				X	X						3
Identification and selection of covariates												–
Report available patient-level, provider-level, facility-level and device-level characteristics	X			X	X					X		4
Capture common co-occurring illnesses	X			X	X					X		4
Differentiate between covariates and confounders	X						X				X	3
Remove irrelevant independent variables from the model	X											1
Missing data												–
Assess the level of missingness	X								X			2
Attempt to determine the type of missingness	X								X			2
Discuss how missing data were handled	X								X			2
Statistical analyses												–
Report and justify the model identification method	X							X	X			3

HTA, health technology assessment; OPC, objective performance criteria.

Engagement of key stakeholders was cited by 38% of the participants as the most important element. The second most frequently cited element was the selection of the data source (23%). The participants cited six possible anticipated challenges during the OPC generation process (table 4). Access to meaningful and high-quality data sources (47%) and reaching stakeholder consensus (25%) were the most frequently mentioned challenges.

Table 4

Challenges to objective performance criteria (OPC) generation identified by stakeholders

Biggest challenge	Registry representative	Health system representative	Clinician	Patient and Patient caregiver	Device manufacturer	Device application reviewers	Data coordinator	Data analyst	Data informatician	Payer/HTA	Total (n=36)*	Percentage
Stakeholder consensus	1	1			2	1	1	1		2	9	25
Meaningful, accessible, high-quality data source	2	2	3	1	1	2	1	1	1	3	17	47
Identifying and measuring meaningful outcomes and covariates				2			1	1			4	11
Identifying medical devices that would benefit from OPC							1	1		1	3	8
Resources: funding, data sources, patient-centerd stakeholders				1						1	2	6
Disseminating the results correctly				1							1	3

*Some stakeholders cited more than one challenge; thus, the total number of cited challenges is greater than the number of stakeholders engaged.

HTA, health technology assessment; OPC, objective performance criteria.

Discussion

Gathered reflections from participants indicate that a framework addressing the generation of OPC within the USA should touch on the following five elements: (1) identification of medical devices for the development of OPC, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings. The elicited subthemes from the participants are consistent with the limited guidance provided by the FDA on OPC and the use of RWE (Real-World Evidence) for medical device evaluation.18 19

Regarding identification of the device for OPC development, the FDA concurs device technology must be sufficiently mature and OPC need to be updated to keep abreast with the evolution of clinical practice and new technologies. FDA guidance acknowledges OPC generation is a collaborative effort and generally cannot be executed by a single stakeholder. It also recommends collaboration with medical or scientific societies or standards organizations to potentially increase the validity of the OPC and engagement with FDA staff prior to study initiation.6 Discussions with participants highlighted additional stakeholder groups to consider for engagement, including: clinicians, patients, data owners, payers, informaticians and hospital health system representatives. Engagement of patients was underscored by several participants reflecting increased attention to patients’ role in product development and assesment.20 21 The importance of the selecting appropriate and high-quality data sources was repeatedly emphasized.18 22 23 Participants indicated prioritization of datasets capturing clinically and meaningful outcomes, as well as linked datasets, such as high-quality registries and claims, should be leveraged to mitigate limitations of individual data sources.

Previous work has identified elements of appropriate statistical analysis for generation of OPC acceptable to all stakeholder groups, including: accounting for statistical uncertainty through sensitivity analyses, identifying prognostic factors of the intended population and capturing clinically meaningful and patient-centerd outcomes.5 In addition to these elements, participants in this project also cited: accounting for missing data, employing methodologies to control for confounders, applying apt regression models and conducting relevant subgroup analyses as components of an appropriate statistical analysis. Determining what constitutes appropriate statistical analysis is challenging given the potential complexity, advantages and disadvantages of various approaches. Transparent reporting of selected methods and dissemination of findings has been a point of focus for many guidelines.24 25 Specific to OPC, participants stated communicated information ought to summarize how OPC were determined, main findings, operationalization of OPC, appropriate use and known limitations of OPC, and implications of findings on clinical, patient and regulatory decision making. Finally, stakeholders noted that if one criterion of the framework cannot be met due to foreseen or unforeseen challenges, then the process of OPC generation needs to be temporarily discontinued until that criterion can be successfully met. For example, if a high-quality, available and appropriate data source is not available to assess selected relevant endpoints then OPC generation may not be possible.

Participant discussions elicited elements that may guide generation of high-quality and widely applicable OPC. The findings encompass a variety of perspectives through the engagement of participants representing a wide variety of stakeholder groups involved in aspects of development and evaluation of medical devices. Furthermore, uses and benefits of OPC vary among stakeholder groups, which may allow the elements to inform the creation of OPC with wider applicability. This study has important limitations that should be noted. Since this study leveraged discussions with key stakeholders, it is important to note that individual stakeholders provided input from their professional perspective. However, this input may not represent the perspective of all stakeholders or all individuals representing the stakeholder. Moreover, discussions were not audiorecorded. Relying on notetaking may introduce recorder bias where the notetaker determines what is important or significant enough to record in the notes. While we attempted to capture all stakeholder types involved in or affected by OPC creation, some stakeholder types indirectly involved, such as device engineers, were not included. Given that limited literature on OPC generation exists, it is important to note that these recommendations may evolve as more OPC studies and clinical area-specific guidance are published.

While previous frameworks and guidance have been tailored to specific clinical areas, discussions with participants in this study elicited foundational elements of a framework that may be generalizable to class III devices requiring OPC as well as adapted to specific clinical areas. This addresses an important gap in the literature and provides suggestions that may be helpful for device development and evaluation in clinical areas where no or little guidance exists. This study identifies five elements that may be considered in a formal framework guiding the creation of OPC within class III medical devices. Further work, including engagement of a large and diverse group of stakeholders with formal consensus building methods, is needed to create a formalized framework on OPC generation. A formalized framework can guide organizations developing clinical area-specific guidance, government agencies providing guidance related to OPC for regulatory purposes and investigators interested in contributing to OPC generation.