ASSESSING THE VALUE OF INNOVATIVE MEDICAL DEVICES AND DIAGNOSTICS: THE IMPORTANCE OF CLEAR AND RELEVANT CLAIMS OF BENEFIT.

OBJECTIVES: Large numbers of new medical devices and diagnostics are developed and health services need to identify which ones offer real advantages. The National Institute for Health and Care Excellence (NICE) has introduced a system for assessing technologies that are often notified by companies, based on claims made for their benefits to patients, the National Health Service, and the environment.
METHODS: Detailed scrutiny of claims made for the benefits of products and the corresponding evidence, seeking associations between these and the selection of products for full evaluation to produce NICE guidance.
RESULTS: Between 2009 and 2015 a NICE committee considered 169 technologies, of which it selected 74 (44 percent) for full evaluation, based on the claims of benefit and the evidence available. An average of 7.5 claims were made per technology; the total number did not influence selection but presence of studies supporting all the claims (p < .001) or any of the claims (p < .05) had a positive influence, as did claims for quicker patient recovery (p < .001). A greater number of studies to support the claims made selection more likely (p < .001), as did cohort studies (p < .05) and surveys (p < .05) but, unexpectedly, not randomized trials. The Medical Device Directive class had no influence.
CONCLUSIONS: This study presents categories of claims that may be useful to those developing new products and to others engaged in health technology assessment. It illustrates the importance of relevant evidence and of having a clear vision of the place of new products in care pathways from an early stage.

Successful development and adoption of medical devices and diagnostics needs to be underpinned by a clear vision of the benefits that they will bring to patients, to health services, and where they will fit into pathways of care. This vision is required from the earliest stage of designing new technologies and it needs to be re-examined, in consultation with those who will use the technology in clinical practice, as development proceeds (1). Without this focus on where and how new technologies will be used, the setting and the endpoints of clinical studies cannot be optimized; and both devices and diagnostics may fail to deliver the benefits and efficiencies that they might otherwise offer.

We and others have previously described a system developed by the National Institute for Health and Care Excellence (NICE) for evaluating and promoting the adoption of new devices and diagnostics (2;3). Both the initial selection process and the subsequent evaluation are informed by claims made by companies about the benefits of their products. These claims are presented as lists of claimed advantages of the new technology over current management (a) for patients, (b) for the health service (specifically including the cost consequences of introducing the new device or diagnostic), and (c) any sustainability benefits (reducing energy use).

Our previous study (3) documented the numbers of technologies that were selected (or not selected) for evaluation to produce NICE guidance for the health service. It listed the reasons given for not selecting 80 of 147 (54 percent) technologies notified by companies, all based on the claims made and the evidence proffered to support them (3).

The claims of benefit are fundamental to NICE's decision making because they inform the scope of any evaluation including the relevant outcomes of interest to patients and health services. Often, insufficient attention is given to the issues that will determine the uptake of their products and their advantages (or lack of advantage) in everyday practice. These may include the setting in which they offer most benefit (for example in primary or secondary care), the most appropriate patient group, the most important outcome measures, and what “current management” to compare them against (i.e., which other treatment or diagnostic modality is most widely used).

The aim of this study was to examine the claims of benefit made for technologies notified to the NICE Medical Technologies Evaluation Programme during its first 6 years. In particular, this study categorizes the types of claims commonly made and it scrutinizes the evidence presented to support the claims. The objective was to focus the attention of those involved in development, research, and marketing of new products, as well as those involved in health technology assessment (HTA), on early and careful consideration of the claims that they plan to make and the evidence behind them.

METHODS

The processes and methods of the NICE Medical Technologies system have been described in detail in previous articles (2;3). In brief, NICE receives notifications about new devices and diagnostic technologies from companies (manufacturers, distributors) and less commonly, clinicians or patients, using a standard form. This explicitly requires a description of the technology and lists of claims for its advantages for patients, the health service, and sustainability, compared against well-defined “current management.”

The claims of benefit, together with summaries of current evidence (which can include peer-reviewed papers, conference abstracts, and other material, including unpublished manuscripts) are presented in a briefing note that is considered by an independent advisory committee with twenty-five members, including doctors, a nurse, scientists, HTA experts, lay people, and a device regulator (hereafter “the committee”). They received advice from experts in the area of the technology and its use, and sometimes from patient representatives. Based on their deliberations, technologies are either selected, or not selected, for further evaluation by NICE, to produce guidance on adoption for the health service in England (4). No information is published on individual products that are not selected, to protect the interests of companies, and we have previously reported the reasons for nonselection (3).

This study examined the claims, and the studies and other information proffered to support them, which underpinned the committee's deliberations about whether or not to select technologies for production of NICE guidance. This was done as follows.

Numbers and Categories of Claims

The number of claims for each notified technology was recorded, both in total and under the headings of (a) patient benefits, (b) health service benefits, and (c) sustainability (saving energy). The principal claims were then categorized by one author (C.J.D.P.) and into specified categories as shown in Tables 1 and 2. The total numbers were calculated of each category of claim associated with technologies being selected or not selected The possibility of confounding variables was considered and rejected on a case-by-case basis in the event that the claims overlapped in the clinical context.

Table 1.

The categories of claims used in this study for patient benefits, health system benefits and sustainability: these are the three types of benefit specified when notifying any technology.

(a) Patient benefits: • Safety (specifically less complications) • Clinical benefit (e.g., more effective, lasts longer) • Experiential benefit (e.g., less time in hospital, less painful, fewer visits) • Psychological benefit (e.g., immediate results, more information) • Quicker recovery (e.g., quicker return activities and/or to work) (b) Health system benefits: • Shorter length of hospital stay • Fewer attendances • Lower “treatment costs” (e.g., consumables, facilities, treatment time) • Less staff • Lower pay grade of staff • Less treatment of complications (c) Sustainability: • Less waste • Less direct use of power • Less travel

Table 2.

Numbers and Categories of Claim

Claim	Selected (n = 74)	Non-selected (n = 95)	Total (n = 169)
Clinical Benefit	192	210	402
Experiential Benefit	46	88	134
Less Staff Resource Use	32	58	90
Fewer Attendances	41	47	88
Lower Treatment Costs	36	50	86
Safety	44	41	85
Less Waste	29	51	80
Less Treatment of Complications	38	35	73
Shorter Length of Stay	29	34	63
Quicker Recovery	23	18	41
Less Direct Use of Power	16	19	35
Less Travel	13	18	31
Psychological Benefit	10	9	19
Less Staff Resource	6	11	17
Total Claims	568	692	1260

Note. Numbers and categories of claim for 169 medical technologies that were either selected or non-selected for further evaluation at NICE by the Medical Technologies Advisory Committee (MTAC). All technologies had multiple claims.

Studies Supporting the Claims

The studies presented to support the various claims were also divided into different types by one of the authors (C.J.D.P.). Empirically, 23 types of study were used and these are listed in Table 2. Each study was assigned to only one type.

Factual or Aspirational Claims

The claims were subdivided into those that were deemed to be factual (e.g., “This technology produces …. does …. is associated with ….”) and those that were judged to be aspirational (e.g., “This technology may …. might …. could possibly ….”).

Opinions of Medical and Scientific Expert Advisers

As part of the selection process, expert advisers complete a detailed questionnaire in which one question is about whether they consider the technology to be: (i) a minor variation on existing technologies with little potential for different outcomes and impact; (ii) a significant modification of an existing technology with real potential for different outcomes and impact; or (iii) thoroughly novel, that is, different in concept and/or design to any existing technology.

Statistical associations were sought between the number of expert advisers opining that a technology was either a significant modification or else thoroughly novel, compared against whether or not the technology was selected. In addition, the presence or absence of evidence from patient groups (which is requested for all technologies, but not always received) was tested for its influence on selection.

Classes of Technologies

Technologies were classified according to the European Medical Devices Directive from information provided by the manufacturer. This classification (classes I, IIa, IIb, or III) is described in detailed legislation (5). Associations were sought between different classes of device and whether or not they were selected.

Statistical Analysis

When descriptors where dichotomous (yes or no answers, e.g., “evidence to support all the claims = yes”), binary logistic regression (SPSS version 22) was used to compare selection or nonselection of technologies. Significance was assigned when p < .05.

When descriptors were continuous (e.g., numbers of claims) logistic regression (SPSS version 22) was used to compare selected versus nonselected technologies. The Chi square statistic was derived. Significance was assigned when p < .05. When comparing the numbers of claims around a sequence of thresholds (e.g., technologies with more than three claims or more than four claims), data were not normally distributed and the Mann-Whitney U-test was used repeatedly to compare the groups below and above each threshold level. Significance was assigned when p < .05.

RESULTS

Between December 2009 and November 2015 the committee considered 169 technologies. It selected seventy-four (44 percent) for evaluation by NICE and decided not to select ninety-five (56 percent).

In total, 1,260 claims of benefit were considered by the committee; a mean of 7.5 claims per technology. The total number of claims was neither significantly associated with selection nor non-selection of technologies (p = .294). Table 2 shows the number of technologies selected and not selected, mapped against the presence of one or more claims in each of the fourteen defined categories. Only one particular category of claim was significantly associated with technologies being selected for evaluation by NICE: that was the benefit to patients of quicker recovery (p < .01). By contrast, claims of experiential benefit for patients were associated with technologies not being selected (p < .01).

Regarding the subdivision of claims into factual or aspirational, aspirational claims were significantly associated with selection (p < .001).

In total, 702 studies of various types were presented to the committee in support of the claims (Table 3). Technologies that were selected had significantly greater total numbers of studies to support the claims than those that were not selected (p < .001). The following types of studies were significantly associated with selection decisions: cohort studies (p < .05), surveys (p < .05), and cost analyses (p < .05). Types of study that were associated with technologies not being selected were pilot studies (p < .05), unpublished studies submitted in confidence (p < .05), and marketing claims with no substantiating evidence (p < .001).

Table 3.

Numbers and Types of Studies Considered

Type of study	Selected (n = 74)	Non-selected (n = 95)	Total (n = 169)
Observational	57	49	106
Randomized Controlled Trial	52	43	95
Prospective	26	49	75
Diagnostic Accuracy	47	27	74
Case Series	30	37	67
Validation	29	25	54
Abstract	11	27	38
Systematic Review	20	9	29
Cohort	19	7	26
Academic in Confidence	2	18	20
Survey	14	4	18
Cost Analysis	13	4	17
Pilot	1	11	12
Case Report	7	5	12
Laboratory	2	8	10
Crossover	7	2	9
Audit	5	4	9
Feasibility	4	5	9
Case Control	3	3	6
Post-market surveillance	1	5	6
Registry	1	3	4
Modelling	1	3	4
Review	2	0	2
Total	354	348	702

Note. Numbers and types of studies considered by the Medical Technologies Assessment Committee (MTAC) for 169 medical technologies either selected or non-selected for full evaluation by NICE.

When there was evidence to support all of the claims, this was significantly associated with selection by the committee (p < .001). The presence of evidence to support any of the claims was also significantly associated with selection (p < .001). Technologies with more than three claims that were supported by studies were more likely to be selected (p < .001, Mann Whitney U-test) than those with three or less claims supported by studies.

There were eight technologies with no studies to support the claims made about them. These were significantly less likely to be selected than those with evidence (p < .001, Mann Whitney U-test). Two technologies with no supporting studies were selected for evaluation but were subsequently either rejected at the next stage of evaluation or were the subject of guidance which stated that the case for adopting them was not supported. The number of experts who described technologies as significant modifications of existing technologies was significantly associated with selection (p = .007), but descriptions of novelty or minor modifications were not. The total number of experts who provided advice was not significantly associated with selection, nor was expert testimony from patients or patient groups.

The class of the technology under the European Medical Devices Directive did not have any correlation with selection for evaluation by NICE. A subgroup analysis was performed comparing implanted, Class III devices (n = 7) against nonimplanted Class III devices (n = 10), and other classes of device. No statistically significant differences were observed in either the levels of evidence or the selection decision based on whether or not a device was implanted.

DISCUSSION

This study was based on a large number and wide range of different medical devices and diagnostics, all with claims of advantages for patients and the health service. Less than half were selected for further evaluation and this study has demonstrated associations between the types of claims and the evidence supporting them versus whether or not technologies were selected. We found no association between the absolute number of claims and the likelihood of selection. When claims are numerous they are often repetitive (sometimes almost identical) and that may bias decision making negatively, compared with a small number of well-conceived and clearly different claims. By contrast, the number of studies supporting the claims was associated with selection.

The fourteen categories of claims reflect our experience of the benefits most often claimed (and desired) for new products. These categories may be useful to industry and to others in considering the range of benefits that technologies can provide. When the committee considers claims, two issues, in particular, influence its decisions. First, what is the comparator? It is not uncommon for companies to use a comparator that is obviously more costly, but that is not the most widely used alternative to their new product; or to assume that a particular form of management is the norm, when clinical practice varies widely.

It can be difficult to establish what “current management” most typically comprises, but investigating that thoroughly at an early stage of product development is of pivotal importance. That means liaison with a variety of clinical experts. The second issue is the setting in which a new technology is promoted. This issue of its “place in the care pathway” is fundamental, perhaps most floridly demonstrated for certain diagnostic tests, which could have a place either in primary or in secondary care. Not infrequently companies choose to focus on one of these, while clinical experts and the committee see a more persuasive value proposition for the other.

The claims of benefit made for technologies gave insight into the restricted vision of some companies into the potential and value of their products. If claims lack sufficient merit and evidential support to persuade NICE's committee to select them, this suggests a lack of consideration throughout the stages of conception, design, and development. An important objective of this study is to influence the thinking of manufacturers toward better planning of the application of their new technologies; this underpins generation of the most appropriate evidence of the value of products to patients and to health services. For many devices, there is a transfer of intellectual property when a larger company buys the rights to a product from a smaller company or a developer. That is a time at which there is a particular risk of disruption to the fluent generation of evidence.

The research evidence available to support devices and diagnostics is typically much more limited than for pharmaceutical products and it is sometimes very poor indeed. Reasons include less regulatory demand for evidence and limited funding for studies: many device companies are small and are limited not only in their financial resources but also in their research experience. Short product lifespan, the tendency of devices to evolve, and the threat of “fast followers” from competitors are other influences. To maximize the available data, the NICE Medical Technologies Programme is prepared to consider a very wide range of types of evidence, as shown in Table 2.

Findings that might have been expected (and that are reassuring) were that selection was associated with technologies that had studies to support all the claims made about them, and also with studies to support any of the claims.

With regard to the types of studies and their influence on selection, randomized controlled trials were not associated with a greater likelihood of selection, based on our statistical analysis. While this might suggest a bias against the normal hierarchy of evidence, it may well be because randomized trials of technologies are sometimes designed with outcomes and endpoints that are not the most relevant for an assessment of their value to patients or the service, or they are designed without the most appropriate comparators. Lack of appropriate comparators in trials may sometimes have been based on choosing ones that are more costly or more likely to prove inferior, rather than the most commonly used alternative; or it may happen because trials were done outside the United Kingdom, where the most commonly used alternative technology is different.

It has been suggested that randomized controlled trials may not be feasible for some implantable medical devices because of ethical considerations associated with such devices (6). However, we did not observe any significant difference between the numbers of randomized controlled trials when we compared class III implanted medical devices and both nonimplanted class III devices or other device classes.

The finding that cohort studies and surveys were associated with selection probably reflects the fact that these are common ways in which devices are studied: they are relatively inexpensive research methods and can accrue information from substantial numbers of patients quite quickly. These observations are not intended to promote them as high quality methods: they simply explain why studies that are low in the normal hierarchy of evidence seem to be influential in this context. Observational studies may address factors such as long-term effectiveness and resource use in ways that are beyond the normal scope of randomized controlled trials.

The concepts of novelty and of promise influence selection decisions (7;8). Getting novel, clever, useful technologies into health care more rapidly was the founding aim of the NICE Medical Technologies Evaluation program. The concept of promise is more difficult to study, but we believe that it is influential and that technologies with “plausible promise” may be selected even when the evidence on them is limited (8). Our results demonstrated that aspirational claims based on the plausibility of a technology would provide the specified benefit/s were associated with selection. This was completely unexpected; we had subdivided the claims into factual and aspirational in anticipation that the reverse would be the case (i.e., aspirational claims would be associated with nonselection). A possible explanation of this surprise finding is that some technologies with aspirational claims were ones in which the committee saw plausible promise.

Health services worldwide are under financial pressure and need new technologies that increase efficiency and that cut costs, in addition to improving outcomes for patients. Successful introduction of technologies, therefore, requires careful consideration of what advantages they will offer for patients, the health service, and the environment; and of what technologies and practices they will replace. This is precisely what the claims studied in this study are about. Focusing research on these issues will provide the data most appropriate to successful HTA and adoption of new devices and diagnostics. Large companies should already have the expertise and connections to address these issues. Smaller companies may benefit from expert HTA-focused consultation on their product development proposals, such as that offered by the NICE Scientific Advice (9).

There have been reports of other initiatives to evaluate medical technologies, since the start of the NICE MTEP program, which have addressed some of the issues discussed above. In particular, Medtecha (10;11) has addressed economic and clinical effectiveness, and Rothery et al. (12) have considered the need for further research on promising technologies where there are gaps in the evidence.

The strengths of this study are unique access to information presented to a NICE medical device committee that would not usually be available to the public, and the capacity to study decisions about a large range and number of technologies over a 5-year interval. The weaknesses of this study include a potential for bias, because NICE staff and committee members were studying work in which they had been personally involved. Every opportunity was taken to ensure that the decision makers in committee (author B.C. was the chair of MTAC) were not involved in the coding or analysis of data.

This study has shown the kinds of claims and supporting evidence that are associated with a successful outcome when medical technologies are scrutinized by a committee dedicated to identifying those with likely real benefits for patients and for health services. A clear vision of the best application for any new technology and a well-crafted value proposition form the foundation for its success. If these have been well planned then the claims about the technology can be presented with confidence and should have sufficient evidence to support them. Those are the keys to success when scrutinized by NICE and other HTA bodies, and to rapid adoption.