Towards early and broad evaluation of innovative surgical devices: integrating evidence synthesis, stakeholder involvement, and health economic modeling into the clinical research stages of the IDEAL framework.

Introduction

New surgical devices are entering healthcare systems at an unprecedented pace.1 Scientific evaluation of such devices is, however, perceived to lag behind that of new drugs.2–4 Although innovation is often seen as a positive development, it is currently hard to predict or establish how innovative devices improve surgical care.5

While new drugs must show substantial evidence of effectiveness and safety through clinical trials, medical device regulations in the European Union and the United States of America have historically focused on proof of safety as a minimum requirement.6 7 In recent years, both regulations have been updated and now demand clinical effectiveness for high-risk devices, but still allow some surgical devices to gain market approval with little or no clinical evidence.8 To improve the quality of research for surgical devices, the IDEAL collaboration has adapted their five-stage evaluation framework (Idea, Development, Exploration, Assessment, and Long-term follow-up) to specifically provide recommendations for clinical studies on innovative devices (IDEAL-D).9 10 The collaboration aims for evaluation that results in rigorous, safe, and fast evidence gathering, using not only clinical studies but also approaches such as stakeholder consultation, modeling, and cost-effectiveness studies.11–13

The recent attention for these other methodological approaches is important, as clinical trials are expensive and the capacity for conducting these trials is limited. Ideally, only devices that have the greatest potential to improve healthcare and are aligned with the needs and beliefs of involved stakeholders are selected for clinical trials.

This requires methods that enable evaluation of surgical devices from a very early phase and allow for an evaluation that is broader than the analysis of clinical effects. Developing and applying such methods has proved to be difficult, as the earliest stages of innovation are characterized by large uncertainty and a lack of evidence. In these situations, it is tempting to delay solid evaluation or to only include direct, tangible, effects. The potential risk is that devices are developed that are not desirable, effective, affordable, or supported by stakeholders. To avoid this risk, methodological approaches are needed that allow for an early and broad assessment. However, little is known about which methodological approaches can be used and how these methodological approaches should be integrated in the IDEAL framework. In this paper, we therefore take a first step towards an iterative approach and show how evidence synthesis, stakeholder involvement, and health economic modeling can be integrated in the IDEAL framework. To make this more tangible, we describe an empirical case study demonstrating our approach in clinical practice. It contains an example of one of our previous projects and the lessons we have learned during this project.

Integrating evidence synthesis, stakeholder involvement, and health economic modeling into the clinical research stages of the IDEAL framework

Element 1: evidence synthesis

Evidence synthesis refers to the process of bringing together information from a range of sources and disciplines to determine what is already known, and is ideally the first step in our approach. It is important to look at current clinical practice, the innovation, and the healthcare system. Hereby, information on the performance of current practice and the healthcare system can be generated, as well as information on innovation characteristics and the existence of other innovative, competing techniques. This provides a solid evidence base for further research and development (R&D) and can avoid developing devices for indications where other innovative technologies are already being developed. Scientific and gray literature are key sources of information. Appraising the quality of the evidence is a crucial part of evidence synthesis, as low-quality studies could misinform further studies or create diverging opinions among stakeholders.14–16 In our empirical example presented in online supplemental materials, we show how our approach was used in the evaluation of sentinel lymph node biopsy for the management of oral cavity squamous cell carcinoma. The evaluation process took off with an evidence synthesis of international guidelines.

Element 2: stakeholder involvement

Stakeholders should be involved in the innovation and evaluation process. It is important to identify a broad range of stakeholders, because novel surgical devices work most optimally when they are valuable for all stakeholders like patients, relatives, and medical professionals.17 18 We employ qualitative methods for stakeholder involvement, for example interviews, because these methods are most suitable to perform an open and detailed analysis of the problems, solutions, knowledge and values that the included stakeholders describe. It is important to acknowledge that stakeholder might consider different domains in characterizing the value of a device and attach different weights to those domains. During the early evaluation, the interests of different stakeholder need to be brought together in the decision-making process. Interaction between stakeholders to carefully balance these interests is therefore crucial to our approach. Stakeholder involvement will contribute to a broad set of criteria that must be fulfilled for the innovation to have added value and important study parameters or outcome measures to be taken into account in next study phases. In our empirical example, interviews were used to determine improvements in oral cavity squamous cell carcinoma care and highlighted different opinions about the safety and effectiveness of innovative sentinel lymph node biopsy.

Element 3: health economic modeling

Health economic modeling is a relatively quick and inexpensive way of exploring the potential consequences of an innovation.19–22 In our approach, we look beyond the traditional use of modeling, because we believe that early-stage models should not be used as a tool for a definitive assessment of an innovation, but have an exploratory function.18 23–26 The first stages of innovation are often characterized by large uncertainty (eg, about innovation effectiveness; how, when and where to implement the innovation in the care pathway). In this stage, exploratory modeling approaches can be used to explore effectiveness gaps in the current clinical pathway. Next, conditions under which an innovation can be of added value can be explored. It should be used to determine what is needed for all relevant perspectives, such as society as a whole (eg, “could this device bring added value at an affordable cost for society”), instead of only adopting a business perspective (eg, “could this device be commercially viable”). Exploratory modeling allows for multiple scenarios or stakeholder views to be modeled and can deal with the complexity and uncertainty associated with innovation. In this way, health economic models can be used to inform study design, as they identify important knowledge gaps in current practice, determine which parameters should be studied, and set out conditions under which the innovation has added value for society.22 23 25 27 In the empirical example, modeling was used to evaluate the cost-effectiveness of multiple innovative strategies and to explore important uncertainties.28 In a next step, the model was adapted to include individual patient characteristics. In this way, the best treatment option for individual patients could be investigated.29

Element 4: clinical research

To establish the actual added value of an innovation in practice and gain regulatory approval, reimbursement, and adoption in clinical practice, clinical research is needed. By using frameworks such as IDEAL and IDEAL-D next to regulatory requirements, clinical research using the appropriate methodology can be set up for each clinical research phase. IDEAL recommends a gradual approach, with stage 0 as preclinical research containing a wide variety of approaches, including the above-mentioned elements, cadaver studies, usability testing, etc. The clinical stages comprise stage 1 as case reports of first-in-human uses of innovations, stage 2a as prospective development studies describing iterative development of the procedure, stage 2b as prospective exploratory studies describing the efficacy of more or less stabilized techniques, stage 3 as randomized controlled trials or equivalent alternatives assessing the comparative effectiveness of the technique against current practice, and stage 4 as registries to study long-term effects.9–11 Our empirical example shows that not all stages of the IDEAL framework were followed. Only recently, sentinel lymph node biopsy in oral cancer was investigated in a randomized controlled trial.30

Integrating the four elements

IDEAL proposes a gradual approach to the clinical evaluation of surgical innovations. Hirst et al already suggest the use of evidence synthesis, stakeholder involvement, and health economic modeling in IDEAL 0.11 The novelty and cornerstone of our approach is that these elements should inform each other and that they should be interwoven in each of the stages of the IDEAL framework, as is displayed in figure 1.

Figure 1

Overview of the iterative approach. Evidence synthesis (top left), stakeholder involvement (top right), health economic modeling (bottom right), and clinical research (bottom left) are illustrated as individual elements which inform each other to improve the innovation process, as depicted in the golden circle. Our approach should be seen as an ongoing, non-linear process that is guided by research, and design questions that occur in the innovation process.

For example, evidence can be used to provide an overview of current clinical practice and the healthcare system, and generate a first overview of the innovation characteristics and potential competing techniques. This information can subsequently be used for stakeholder deliberations to determine where in the system the innovation should be placed and what study outcome measures are relevant. These outcome measures can subsequently be incorporated in health economic models. Health economic models explore the room for improvement in current practice, under which circumstances the innovation may provide better outcomes than standard practice and what parameters cause relevant uncertainty and should be studied. All this information can be used to design and conduct a clinical study within the IDEAL framework. Thereafter, findings from these clinical studies form the basis of further evaluation and can be used to update the evidence synthesis, inform stakeholder deliberation, and update health economic models.

We argue that in each innovation proces, all elements should be addressed. In this way, a broad scope is ensured which increases the chances of valuable innovation. However, to really steer R&D of new surgical devices towards real added value to patients, healthcare and society, an early starting point, and multiple iterations of the approach are probably necessary, as we have seen in previous attempts of our approach (online supplemental materials). We think that our approach should start as early as possible, with evidence synthesis. Thereafter, elements can be iterated in a non-linear fashion, where unresolved research questions from former elements are used to determine which element or elements are most appropriate for the next phase. This results in an ongoing process throughout the stages of the IDEAL framework where all elements are iterated.

Discussion

Thorough early evaluations can help to optimize innovative surgical devices during the innovation process. We believe that by integrating evidence synthesis, stakeholder involvement, health economic modeling, and clinical research, it is possible to overcome the challenges related to the evaluation of surgical devices. Using this combination of elements allows to start as early as an unmet need is detected or an innovative idea arises, and take a broad scope during evaluation. This will improve the current evaluation frameworks that are available.

Some remaining methodological issues need to be addressed. First, the suggested approach could initially increase the workload of the already complex innovation process. It is important that the recommended methodologies are accessible and usable by innovators, researchers, and other potential end users. That is why they should closely collaborate, because they hold the experience and knowledge that is needed to make the methods usable and valuable in the innovation pathway.18 In the end, we envision that by using our approach the innovation pathway will be shortened and smoothed, as problems and challenges will be detected in an early phase and can therefore be resolved during the design phase rather than at any later stage when it will be harder to make design changes. Second, the innovation process is often hard to plan in advance, it is probable that the starting point, the number of iterations, and the choice of elements depend on the context and problems of the specific innovation process. At least all elements should be addressed once, but for a thorough evaluation, an early starting point and multiple iterations of the approach are probably necessary. Third, some elements could also benefit from further methodological developments to make them more applicable for innovative surgical devices. For example, there is a need for methods of stakeholder involvement that are feasible and flexible, but still allow for a thorough elicitation of needs and values, as well as health economic modeling methods that address specific challenges related to surgical devices, such as learning curves.

In view of the enormous influx of new surgical devices and their potential consequences, both good and bad, we all have the responsibility to properly evaluate these innovations. In the first stages of the IDEAL framework, methods such as stakeholder involvement and health economic modeling were already mentioned, but no recommendations were made on how to perform and apply these methods.11 12 We have already explored how these methods can be used in the early stages of innovation, and aim to further develop these methodologies so that they can be integrated into the various stages of the IDEAL framework.31 32

By combining the four elements, it is possible to start early and take a broad scope during evaluation. Ultimately, we hope that by using such an iterative approach throughout the innovation process, it will contribute to desirable, effective, and affordable surgical devices.