New guidelines for research in airway device evaluation: time for an updated approach (ADEPT-2) to the Difficult Airway Society's 'ADEPT' strategy?

In this article we present the learning from a clinical study of airway device evaluation, conducted under the framework of the Difficult Airway Society (DAS, UK) 'ADEPT' (airway device evaluation project team) strategy. We recommend a change in emphasis from small scale randomised controlled trials conducted as research, to larger-scale observational, post-marketing evaluation audits as a way of obtaining more meaningful information.

Introduction

In this article we present the learning from a clinical study of airway device evaluation, conducted under the framework of the Difficult Airway Society (DAS, UK) ‘ADEPT’ (airway device evaluation project team) strategy. We recommend a change in emphasis from small scale randomised controlled trials conducted as research, to larger-scale observational, post-marketing evaluation audits as a way of obtaining more meaningful information.

Exactly ten years ago, DAS published the output of its ADEPT strategy [1]. The initiative was primarily designed to address the then-growing concern that airway devices, especially supraglottic airway devices (SADs), were being introduced to practice without any clinical trial evidence of efficacy. This was happening because, while the Medical Devices Directive (MDD) regulations required various technical specifications to be met before devices could be legally marketed in Europe (after CE-marking; the mark assigned to a device after meeting minimum regulatory requirements), there was no necessity for peer-reviewed clinical trial data, as there is with drugs. This mirrors the regulatory framework in North America and Australasia [2]. In extreme scenarios, there were reports of airway tubing that had virtually no lumen, or that easily disintegrated on use causing severe leaks from breathing systems [3]. Novel SADs performed poorly when subjected to clinical trial [4] and new single-use bougies were later found to be inferior to existing gum elastic ones [5].

Attempts before ADEPT included the suggestion of Cook and others of a three-step process. In a first stage, manikin studies could exclude major ‘obvious’ issues; a second stage of a rigorous pilot patient study could exclude major safety issues and ‘exclude inefficacy’; and a third stage a randomised controlled trial (RCT) would compare the new device against the current ‘gold standard’ device [6, 7]. However, these proposals had gained no traction with manufacturers or regulators. Instead, ADEPT adopted the new approach of addressing its guidance to purchasers (i.e., hospitals on whose procurement committees would be anaesthetic representation). The idea was that while the MDD could regulate what could be legally sold, this itself did not compel anyone to purchase the product.

The ADEPT guidance specified a level of evidence that professionals, familiar with evidence-based medicine, should regard as the very minimum before a purchasing decision should be made. This was intentionally set quite low, at ‘Level 3b’ in the Centre of Evidence Based Medicine’s hierarchy; i.e., a case–control or historical-control (observational) study (see: http://www.cebm.net/). This was not viewed as the optimum or desirable level, but as something that ought to be readily achievable. Later publications helped define the suitable case numbers for powering studies of this nature [8]. It was noted at the time that very few new devices had achieved even this very modest Level 3b evidence base before the marketing stage [1].

Thus, ADEPT helped clarify the roles of the different players in the system. Manufacturers develop and seek to introduce the new device. Regulators grant it the appropriate certificate for marketing (e.g., in Europe the CE mark) after it meets minimum technical (but not clinical) standards. Post-marketing, clinicians should test the devices in trials (but as we discuss below this infrequently happens) and help identify the better/worse devices or clarify the niche for each device. Journals publish the research. Clinicians also use the evidence to influence purchasing decisions in their hospital, completing the ideal loop.

A second feature of ADEPT was linking this approach to a wider strategy around the development and support of academic anaesthesia. If a certain level of evidence (Level 3b) was now required to achieve sales, the logic was that manufacturers would need to reach out to research-active units (‘DAS trial centres’) to undertake the necessary studies. With DAS acting as co-ordinator, the associated funding recommendations were designed to strengthen the specialty’s academic base. In the UK, the Research Excellence Framework (REF; and its predecessor the Research Assessment Exercise, RAE) is a cyclical national exercise that assesses universities in part by their total grant income; this ranking is then used to allocate infrastructure funding from the respective Higher Education Funding Councils (HEFCs) of the devolved UK governments. In turn, universities run similar in-house exercise for their departments, allocating most support to the strongest, and closing down the weakest or merging these with others. Anaesthetic departments that had hitherto received free boxes of airway equipment counted little in this exercise [10]. To rectify this, ADEPT argued that manufacturers instead provide funding as grant income, to purchase the trial equipment; this funding then credited against anaesthetic department’s REF metrics. Monies could also cover academic time for consultants or nurse or assistant support and other indirect and overhead costs. This aspect was strategically important: previously in the pre-2003 ‘old’ consultant contract, research had been conducted within a consultant’s job plan, essentially for free (i.e., the NHS underwriting all the time costs), with the manufacturer providing equipment without charge. The new 2003 consultant contract defined ‘supporting professional activity’ (SPA) as the time to undertake such work within consultant job plans, but over the years the initial amount awarded has been reduced from 2.5 SPA (10 h) in many Trusts to a core of just one (4 h), and moreover, it was specified that research had to be directly funded via identifiable income streams [11-13].

Has ADEPT worked as envisaged by its authors?

It is difficult to disentangle the impact of ADEPT from the wider issues concerning the state of UK academic anaesthesia, on whose strength ADEPT’s success ultimately depends [10, 14]. Ahmad et al. report that there was indeed an increase in the number of airway research trials after ADEPT’s publication, although not necessarily directly due to manufacturers partnering DAS trial centres in the manner outlined above [15]. We now know that there has been a further decline in the number of UK academic centres (only 11 left) and therefore also the number of individual researchers (< 800 anaesthetists have published any paper over three years) [16]. We discuss the direct relevance of this further below. Thus, only 139 clinical trial papers emanated from the UK between 2017 and 2020 in the main anaesthesia journals. Between 2014 and 2016, when DAS conference papers were last published, just 7 of 86 abstracts pertained to clinical trials or case series [17]. The decline in academic capacity across the UK has therefore imposed a severe constraint on ADEPT’s ability to deliver on clinical trials as originally envisaged, and this reality should be factored into any updated ADEPT strategy.

Lessons for a Revised Strategy (ADEPT-2) from the ‘ADEPT-1 Trial’

Accompanying this paper we present the results of an observational trial on a new SAD, conducted precisely along the lines envisaged by the original ADEPT strategy (‘ADEPT-1’) [18]. The lessons we learned led us to propose a revised approach.

The study was multi-centre but was initially planned to include up to five sites. However, only two of these (Dublin and Oxford) were able to progress with their ethics and Research and Development (R&D) applications to any degree before the Covid-19 pandemic caused further delay. At least part of the reason for the lack of progress was research inexperience: two putative centres had only published one paper each in the previous three years and may have lacked the capacity to make progress. Another centre had once been a discrete university academic department but identified as having lost this status with the departure of key research-capable staff [16].

Both Brexit and the Covid-19 pandemic created their own difficulties. The first led to delays in creating international purchase orders. The move to tracheal intubation versus SADs (at least in the first half of the pandemic, to minimise aerosol generation) [19], followed then by formal suspension of non-Covid investigative studies across the UK delayed our progress.

We fortunately (narrowly) avoided involvement with a clinical trial unit (CTU). Clinical trial units are now a big business, viewed by universities as a means maximise income as well as answer research questions. They are well-staffed, specialised entities assisting with study design and trial co-ordination, and increasingly regarded as central to the quality, credibility and impact of clinical research. However, it has also been reported that CTU costs on grant applications vary in ways that do not appear to correlate with the complexity or size of study designs [20]. Researchers submitted an identical hypothetical study of modest size (125 patients) and discovered CTUs’ quoted costs inexplicably ranged from £661,000 to over £1 million [20]. Several years ago, in an unsuccessful DAS-led multi-centre grant application to the National Institute of Academic Anaesthesia (NIAA), we had similarly discovered CTU costs represented > 50% of the total study costs. The same was the case when we approached a local CTU in the early phase of the accompanying study [18]. That said, one potential benefit of a CTU is that it would have aided the ‘adoption’ of this study onto the local clinical research network (LCRN) [21]. These are entities through which the National Institutes of Health Research (NIHR) supports clinical research through research nurses and other facilities, primarily to help recruitment; adding value at no extra cost. While this is undoubtedly of benefit to large multi-centre trials, the advantage to studies in anaesthesia is questionable since both the recruitment and data collection and management is most easily undertaken by the anaesthetists themselves performing the study.

The role of ethics and study design in the new approach

We sought and obtained full ethical review for our study, but now we question whether this was necessary. Registering the work as an audit would have avoided much burdensome bureaucracy and delay, especially since we discovered this had to be duplicated at each of the five planned centres’ R&D units. Notably, the device could have been used, even if a patient had refused consent to participate in the study. Patients were not being consented for the device, which was CE-marked; rather, only for the inclusion of their data in a publication.

Three factors are considered when classifying studies as ‘research’ (which then mandates full ethical review and the inevitable paperwork), two of which are straightforward: (i) the inclusion of anything in the planned interventions or monitoring that departs from standard clinical care (e.g., using non-CE marked equipment); and (ii) randomisation. The third factor is less easy, if not impossible to define: ‘generalisability of results’. Conventionally this refers to the investigators’ intent that their results should have wide application. It does not refer to how others might use the data in a generalised way, which is something outside the control of the investigators. We argue that, in fact, many if not all airway device evaluation studies are ‘technical notes’ containing relevant information, rather than intrinsically generalisable.

Double-blinded, randomized controlled trials (RCTs) are regarded as a ‘gold standard’ but this oft-repeated mantra is wrong. As Rawlins (a former Chair of the National Institute of Health and Care Excellence, NICE) noted: “Awarding such [gold standard] prominence to… RCTs…is unreasonable” [22], and went on to cite Bradford Hill, an architect of RCTs: “any belief that [RCTs are] the only way would mean, not that the pendulum had swung too far, but that it had come right off the hook” [23]. The ‘gold standard’ study design for any question of interest is simply that which answers the question. Sometimes, this is an RCT, but often not. The strength of RCTs is best seen where the intervention can be blinded and where the effect of randomization is not influenced by operator skills or preference. The physical act of writing a prescription requires no skill (other than literacy) and if moreover, the decision-to-treat is generated by an algorithm then even literacy is not required: such a trial is ideally suited to an RCT design. Note that the act of randomizing a patient goes hand in hand with randomizing a practitioner to deliver the given intervention. Randomising a practitioner to administer treatment X when in fact they have more experience with Y is enhancing bias, not reducing it (especially as ‘experience’ cannot be quantified). This bias is not negated by another practitioner elsewhere in the study being randomised in the other direction [24].

An RCT only assesses the ‘average’ effect of intervention in the ‘average’ patient being treated in a random manner by the ‘average’ anaesthetist. Comparing two groups, these ‘averages’ all purportedly cancel out and all we should be left with is the isolated effect of the intervention [25]. In airway management studies, however, averages are rarely if ever relevant. Rather, we are more interested in technical information about how a device performs in certain (ideally ‘real world’) circumstances, at its best and worst, in the hands of representative anaesthetists. The wider interest is to use the ‘technical information’ provided by a previous study and—different from generalising these results—using that data to assess the niche of a device in one’s own practice or locality. The relevance of any pragmatic comparison between for example, a McGrath and a C-MAC video laryngoscope is not that we might intubate faster with one versus the other on average, but simply a pragmatic one that one device has a stand-apart screen which might make it more amenable to team-based intubations; whereas the other has an in-built screen which might make it, say, more amenable to emergency or off-site intubations. This latter utility may not be amenable to quantitative measurement or statistical comparison.

There are also trade-offs between different endpoints that are not captured in an RCT. Is it better to employ a device that helps intubate the trachea on average a few seconds faster, or one that yields a slightly higher mean rate of first-attempt success? Connected with this is the lack of consensus on the most relevant endpoint (not a problem with, say, studies of anti-hypertensive drugs). Studies variously employ time to successful end-tidal CO2 after device insertion; first-placement success rate; time to desaturation; ratio of inspired to expired tidal volume, and others—but ultimately these are all surrogate measures of ‘utility’, where utility is defined by the specific needs of the practitioner at a given point in time. This is the reason why in airway studies there can be no real intent to generalise the results.

The answers to most relevant questions of interest to the airway management community are simply not amenable to a head-to-head RCT trial design. Instead, several single-arm observational data from several centres—perhaps even subjected to meta-analysis—could create a more meaningful and holistic impression of what is relevant to everyday practice [26]. This new ‘gold standard’ has the standing of a ‘technical note’ rather than a ‘research investigation’ and so normally this would not require full ethical permissions [27].

An even more important consideration than positive outcomes in airway research is the rate of complications, side-effects or other limitations that arise with the use of a device, and RCTs are famously poor at identifying these [24, 25]. Even the analysis of secondary or tertiary outcomes for statistical significance is discouraged in an RCT. Probably the best route to assessing complications is the interrogation of large databases, or large-scale observational studies. Examples include the NAP4 study [28]. Also, NAP5 whose focus was primarily ‘accidental awareness’ in fact had many lessons for airway management, with the central roles of neuromuscular blockade especially at induction and emergence reminding practitioners of the need to balance paralysis with concomitant hypnotic (e.g., during prolonged attempts at tracheal intubation) [29]. The large Danish database studies have generated vital information about managing mask ventilation and difficult airway prediction [30]. Therefore, systematic reviews or meta-analyses of several ‘technical notes’ are better placed to describe complications and limitations than are RCTs.

A key strategic advantage to moving away from RCTs is that there is less reliance on strong academic centres with the capacity to deliver these trials. Instead, as Yeung and Shelton recently observed “academic anaesthesia…belongs to us all” [31]. In the study designs we believe more relevant to airway research (Table 1), all anaesthetists can more readily participate.

Table 1

Some non-research study designs are especially relevant in airway management research

Observational	Taking routine measurements during the course of normal clinical practice
Database	Retrospective or prospective analysis of anonymized patient records of outcomes of use of devices/techniques
Before/after	Non-randomised studies where data collected before the adoption of a new device/technique vs after is compared
Audit	Assessing outcomes against a local or national standard of care
Adverse incident	Similar to the database; the monitoring of adverse end-points with routine use of device or adoption of practice
Case series	Description of outcomes after use of device/technique
Best A vs best B	Non-randomised comparison of performance or outcomes of a group of doctors who adopt method/device A vs B
Post-market surveillance	Publishing the routine data after a device is brought to market
Survey of users	Professional surveys of user experience or preferences
Mathematical modelling	Secondary mathematical or statistical analysis of original data derived from patients including systematic review and meta-analysis
Operations management	The study of patient flows; e.g., duration of surgery, time to perform tasks

Conclusions

Our experiences from our trial [18] and the considerations above lead us to recommend that DAS updates the principles of ADEPT. The new approach (ADEPT-2) we believe should incorporate the following:

Airway devices should have some published, peer-reviewed evidence before they are purchased for use (if purchased with no published evidence, Trusts could be increasing institutional litigation risk: CE-marking alone may not be enough);

The most relevant study designs for airway-related research (in CE-marked devices) are generally large scale, single-arm, observational trials where results are published as ‘technical notes’ of performance. Such study designs do not normally need full ethical review and should be classed as audits of practice (albeit adhering to appropriate ethical standards in the collection and storage of anonymised data);

DAS, or other national organisations, should consider creating central databases where performance results could be entered in a continuous manner;

In turn, a large number of published technical notes (or intermittent database interrogation) should be synthesised through continuous, updated systematic reviews or meta-analyses;

The publication of airway-related device technical notes, or other study design results may require dedicated journals (as these papers may be viewed as low impact by conventional journals), and DAS could consider sponsorship of a journal or website (e.g., like an arXiv; see https://arxiv.org/) to facilitate the rapid, open sharing of relevant information.

Our proposals retain the original spirit of ADEPT, with the emphasis on the need for evidence. However, we urge a move away from RCTs unless specifically indicated or justified. Table 1 summarises alternative approaches to RCTs and Table 2 the ethical issues to address in all these alternative study designs. We avoid the previous emphasis on funding streams and therefore better recognise the changed research and publishing landscape, with the decline in academic anaesthesia capacity. Our proposals should encourage more meaningful trial designs to be conducted at scale and pace and at even lower overall cost. The challenge, as ever, is now for airway specialists, anaesthetists, DAS and manufacturers to work together to increase the evidence base supporting use of already-marketed airway devices.

Table 2

Some principles that should guide the ethical conduct of non-research studies as in Table 1

Intent	The primary intent should be to generate data that informs local policies or decision-making, or as technical notes, and not producing generalizable results
Consent	Investigators should make clear where possible that data collected in the course of routine clinical care may be used anonymously in a publication, and seek consent for this use of data
Database	The analysis of databases or retrospective analysis of datasets should ensure non-identifiability of participants (and the databases themselves should have adhered to relevant national ‘general data protection, GDPR’ regulations)
Novel techniques	Where planned in advance, consent should be sought for potential publication of non-identifiable data or descriptions of techniques that emanate from their application (see above); proposed techniques should be simple enough that permit them to be described in plain language to a patient; there should be no additional risks of the planned technique. Trusts usually have a specific policy for the introduction of novel techniques into local practice