Assessing Technologies for COVID-19: What are the Challenges for Health Technology Assessment Agencies? Findings From a Survey and Roundtable Workshop.

BACKGROUND: To date, health technology assessment (HTA) agencies have not been at the forefront of decision making regarding the adoption of interventions for coronavirus disease 2019 (COVID-19). Instead, policymakers have prioritised rapid action in response to the pandemic emergency, with no assessment of value for money. As COVID-19 vaccination coverage increases and healthcare systems begin to recover, HTA agencies will be expected to assess technologies for COVID-19.
OBJECTIVE: We aimed to identify the key challenges when assessing therapeutic and diagnostic technologies for COVID-19, from the perspective of HTA agencies, and identify whether there is a case for novel HTA methods and/or processes to address them.
METHODS: We used a mixed-methods approach, by conducting an online survey of HTA agencies, to collect data about the challenges faced when assessing or planning to assess diagnostic and therapeutic technologies for COVID-19. The online survey was followed by a 'roundtable' workshop of HTA agencies' representatives to discuss the results and to elaborate on their responses.
RESULTS: We received 21 completed surveys (response rate of 45%) and 11 of the respondents joined the roundtable discussion. Five themes emerged from the responses: assessing clinical effectiveness (44%), assessing cost effectiveness (19%), practical (19%), political (11%), and decision making (11%) challenges. At the roundtable, attendees elaborated on the challenges and identified two additional themes: how HTA agencies have responded to the pandemic to date, and how their role might change over time.
CONCLUSION: HTA agencies face both methodological and logistical challenges when assessing or planning to assess technologies for COVID-19. An interim best-practice HTA framework to address the key challenges would be valuable.

Key Points for Decision Makers

Introduction

Since early 2020, the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus and its associated disease (coronavirus disease 2019 [COVID-19]) pandemic, characterised by massive, successive surges in infections, have presented unprecedented challenges for healthcare systems and wider economies. Governments have prioritised acting rapidly to reduce transmission and identify and treat cases, with focus on public health interventions, vaccinations, testing strategies, and treatments for confirmed cases. Health technology assessment (HTA) agencies have continued to function during the pandemic. However, to date, reimbursement decisions for COVID-19 interventions have largely been made based on clinical trial results alone, without a full HTA of their clinical and cost effectiveness [1, 2]. As a result, various diagnostic tests (polymerase chain reaction [PCR], rapid lateral flow) and treatments (corticosteroids, Janus kinase [JAK] inhibitors, remdesivir, monoclonal antibodies) have entered clinical practice based on efficacy data alone, with some later proven to lack clinical benefit (hydroxychloroquine [3]).

This response to an infectious disease crisis may be justifiable, where the ‘rule of rescue’ prevails [4]. Thorough HTA requires the dual luxury of evidence and time, both of which were in short supply during the early part of the pandemic. Furthermore, with governments of high-income countries appearing to sign blank cheques in their efforts to tackle the emergency, the very reason for HTA, i.e. to identify the most efficient allocation of scarce healthcare resources, becomes less important. It may have also indicated a perceived lack of flexibility in usual HTA processes, with policymakers uncertain about how HTA agencies could inform their rapid response.

It is hoped that vaccination programmes will help to bring the urgent threat to healthcare systems under control. It is then likely that public expenditure on technologies to diagnose or treat COVID-19 will come under increased scrutiny. HTA agencies are best placed to inform which options offer the biggest health benefits relative to their cost. Therefore, it is likely that they will be asked to assess novel COVID-19 technologies and potentially review those that bypassed HTA at the height of the pandemic. These assessments are unlikely to be straightforward [5, 6] and may require innovative methods or processes to be developed [7]. It is important to understand the difficulties HTA agencies are likely to face in this intermediate pandemic recovery phase, to help identify how they can add the most value to COVID-19 decision making. Therefore, as part of the Next Generation Health Technology Assessment (HTx) project, we sought to understand the key challenges in assessing technologies for COVID-19 from the perspective of HTA agencies, and identify whether there is a case for developing innovative methods to inform COVID-19 reimbursement decisions. HTx is a Horizon 2020 project supported by the European Union, lasting for 5 years from January 2019. Its main aim is to create a framework for the next generation of HTA to support patient-centered, societally oriented, real-time decision making on access to and reimbursement for health technologies throughout Europe.

Methods

A mixed-methods approach was used with the aim of identifying the key challenges that HTA agencies are currently facing, or expect to face, when assessing technologies for COVID-19. First, the online survey questions were drafted by two researchers (DD and JE) and refined using input from three experienced HTA researchers. The draft survey was piloted with two HTA analysts to assess its face and content validity and completion time. Survey questions are provided in the Electronic Supplementary Material (ESM). The survey was disseminated among both European and non-European HTA agencies. The European HTA agencies were identified from the European Network for Health Technology Assessment (EUnetHTA) website [8] and those approached by another recent survey exploring complex technology assessments [9]. A purposive sample of non-European agencies, with well-established HTA processes or which had already engaged in COVID-19 assessments, was also identified through the International Network of Agencies for Health Technology Assessment (INAHTA) website. Recipients were asked to provide one response on behalf of their HTA agency. A Microsoft Word version (Microsoft Corporation, Redmond, WA, USA) of the survey was shared by email to facilitate consultation between colleagues within the agencies. The survey was ‘live’ for 4 weeks from 30 March 2021. Second, respondents were invited to attend a subsequent roundtable, held virtually on 24 May 2021. Participants were presented with the aggregate survey results, then asked to elaborate on the identified challenges in an open discussion and reach consensus about the most important challenges.

Quantitative data were analysed using descriptive statistics (counts and percentages). Qualitative data, including responses to the open-ended questions and the transcript from the HTA roundtable, were thematically analysed using the Framework Approach [10]. The analysis framework was drafted based on themes that emerged from the survey responses, and refined based on findings from the roundtable, where participants elaborated on the challenges in more detail. Survey responses and the roundtable transcript were coded manually by JE.

Results

Descriptive Analysis

The survey was distributed to 47 HTA agencies (see the ESM) and was completed by 21 (45%) agencies. Characteristics of respondent agencies, including details of their assessments for COVID-19 technologies, are summarised in Table 1. The roundtable was attended by respondents from 11 of the 21 responding agencies (53%).

Table 1

Characteristics of HTA agencies that responded to the survey and attended the roundtable (geographic location and COVID-19 assessments)

Characteristic	Survey respondents [n (%)]	Roundtable participants [n (%)]
Geographical location
Europe	16 (76)	7 (64)
Central and Eastern Europea	3 (14)	2 (18)
Northern Europea	4 (19)	0 (0)
Southern Europea	1 (5)	1 (9)
Western Europea	8 (38)	4 (36)
Asia	1 (5)	1 (9)
Australia and New Zealand	2 (10)	1 (9)
North America	2 (10)	2 (18)
South America	0 (0)	0 (0)
Africa	0 (0)	0 (0)
Agency has assessed COVID-19 technologies
Yes	13 (62)	8 (73)
No	7 (38)	3 (27)
Of those who answered ‘yes’ above
Agency has adapted its methods to assess COVID-19 technologies
Yes	10 (77)	7 (87.5)
No	3 (23)	1 (22.5)
Agency’s assessments (will) include cost effectiveness
Yes	5 (38)	4 (50)
No	8 (62)	4 (50)
Agency will assess the following types of technology
Diagnostics	6 (46)	5 (62.5)
Non-pharmacological interventions	6 (46)	4 (50)
Pharmaceuticals	11 (85)	8 (100)
Vaccines	3 (23)	3 (37.5)

HTA health technology assessment, COVID-19 coronavirus disease 2019

aEuropean subregions according to EuroVoc [11]

Thematic Analysis

Survey respondents were asked to list the perceived challenges for conducting HTA of COVID-19 technologies relating to each of the following: assessing clinical effectiveness, assessing cost effectiveness, and any other challenges (up to five per category). Results of the thematic analysis of the survey responses were presented during the roundtable for discussion.

The responses received were coded and grouped into themes and subthemes. Five broad themes were identified from the survey responses: clinical-effectiveness issues, cost-effectiveness issues, decision making, practical issues and political issues. Two additional themes were identified from the roundtable discussion: responding to the challenges, and the changing role and contribution of HTA.

The most frequently reported theme in the survey responses was clinical effectiveness challenges (Fig. 1). The five main themes included 17 first-level subthemes. The most common were the internal validity and short-term duration of clinical effectiveness evidence, and practical challenges regarding workload and capacity. These could be further divided into 49 second-level subthemes, which are provided in the final analysis framework (see the ESM).

Fig. 1

Themes and first-level subthemes identified from the survey responses. HTA health technology assessment

Assessing Clinical Effectiveness

Challenges relating to assessing the clinical effectiveness of COVID-19 technologies were the most frequently reported theme. Issues associated with the internal validity of evidence were commonly reported (30% of all clinical effectiveness issues), namely a lack of peer-reviewed evidence; a lack of randomised, comparative evidence; small and underpowered study samples; and a high risk of confounding factors. These issues limit the reliability of the clinical evidence base and increase uncertainty.

The clinical effectiveness responses relating to the short-term evidence base noted that reimbursement decisions were being made based on interim study findings or trials with short-term follow-up periods. Additionally, the HTA process typically assesses the evidence base at a given point in time, meaning the available evidence may quickly become out of date as scientific understanding, clinical practice, and the COVID-19 disease rapidly evolve. A general lack of suitable evidence for decision making was reported and challenges associated with the study endpoints, particularly the use of surrogate outcome measures in clinical trials, were collectively reported in almost one-quarter of responses. Attendees at the roundtable discussion recognised that the clinical evidence base, lacking in robustness and reliability, placed HTA agencies in a more uncertain position than usual, making assessments and decision making more difficult.

One participant suggested that many limitations stem from early COVID-19 studies reporting their results in the context of ‘the changing nature of the disease and our understanding of the disease’, and quickly becoming outdated. They suggested that real-world evidence (RWE) could be considered more openly in HTA decision making to resolve this, if such data can provide useful insights quickly and robustly. Another attendee gave their experience of how different study types had been considered over the course of the pandemic, describing a ‘quite astounding shift in what evidence clinicians were willing to accept at the beginning of the pandemic versus a year on’, and cited the example of hydroxychloroquine, which was initially recommended but subsequently removed from guidance in some countries in response to higher-quality evidence [3].

Responses related to the external validity of evidence covered issues relating to heterogeneous study populations, settings, designs, and control arms. Respondents also noted that there are some conflicting trial results. In usual circumstances, this could be addressed using formal evidence synthesis techniques, but the level of heterogeneity limits the usefulness of meta-analysis. At the roundtable, a participant explained that people being treated in hospital for COVID-19 at that time were different to those treated a year previously, who were, on average, older but had since been prioritised for vaccination programmes. Another participant advised that treatment is increasingly taking place outside the hospital setting. HTA agencies should expect available treatment settings and the composition of patients within them to continue to change over time.

Assessing Cost Effectiveness

Of the challenges that could be grouped under the broad theme of assessing cost effectiveness, almost half related to the lack of suitable evidence with which to conduct a robust economic evaluation. A wide range of model inputs was cited in this regard: resource use data, quality-of-life data, long-term outcomes, and downstream care, all of which may require assumptions that reduce the generalisability of an evaluation. Just over one-third of all respondents also cited various technical challenges in developing a de novo decision-analytic model. These included the need to model complex transmission dynamics and system effects, such as hospital capacity, and developing a model structure with an incomplete understanding of the disease.

A number of responses considered what analysis perspective is the most relevant for an economic evaluation of a COVID-19 technology. Many HTA agencies typically take a payer perspective, focusing on health and healthcare effects only [12], but the ‘scale and diversity of benefits a successful treatment could have’ mean some may be considering whether a broader, societal perspective is appropriate. Opposing views on this point were raised at the roundtable discussion. One participant agreed that the analysis perspective is a pertinent consideration due to the massive economic impact of the pandemic and its bearing on non-health sectors. Others warned against setting methodological precedents in response to the pandemic, suggesting that in the long term there will be no clear reason to assess technologies for COVID-19 differently to those for other conditions, and HTA agencies should reflect carefully on the implications of such decisions for future assessments.

Decision Making

A number of responses described challenges related to how all forms of evidence are ultimately used to inform decision making. Five (3% of all reported challenges) questioned the usefulness of an economic evaluation at this time. First, time-consuming economic evaluation may be less relevant to decision makers in the context of a pandemic, who need to act quickly, particularly in the case of some low-cost technologies being repurposed for COVID-19, such as corticosteroids. The results of a cost-effectiveness analysis are also likely to be less informative when key data to inform model inputs are scarce and uncertain. A roundtable attendee commented that there are circumstances where it may be more reasonable to not assess cost effectiveness, such as if pharmaceutical companies were acting ‘with altruism’ in response to an emergency by charging low prices [13].

Second, some of the responses indicated it is difficult to identify the appropriate comparator with current practice evolving rapidly and the likely rapid entry of new technologies. If the comparator does not reflect current practice, then the true clinical effectiveness may be highly uncertain or unknown, and its cost-effectiveness estimate will not truly reflect its value for money.

Third, even if an economic evaluation is considered worthwhile, and a suitable comparator has been identified, the appropriate cost-effectiveness threshold remains uncertain. Survey responses noted there may be a ‘different policy imperative and societal value for successful treatments’ in the context of a pandemic. Almost two-thirds of the roundtable HTA attendees indicated that special consideration should be given to COVID-19 technologies when assessing their value for money, although several elaborated that ‘yes, in some circumstances’ more accurately reflects their view. For example, some explained that the threshold may be implicitly variable over the course of a pandemic, with treatments likely to be judged more leniently when there are no existing treatment options, or if the health burden is more critical, such as during the peak of an infection surge. Another explained that a lower threshold may be appropriate to manage risk given the highly uncertain evidence base, but, conversely, a higher threshold may incentivise rapid innovation and novel treatments for COVID-19.

Practical Issues

Survey responses indicated that the pandemic has imposed difficult practical challenges for HTA agencies and their usual processes. Most of these were categorised as issues associated with workload; specifically, the need to act rapidly and the strain on capacity imposed by an evolving evidence base, disease and clinical practice. Some reported that resources had been diverted towards COVID-19 assessments, at the expense of other routine activities. At the roundtable, a participant explained that their agency had expedited some non-COVID assessments as a result of the pandemic. Specifically, it had done this for a new cystic fibrosis treatment in response to pressure from the clinical community, ‘to ensure that [patients’] health status was better with [access to] the new treatment’, making this high-risk group less likely to become critically ill due to COVID-19.

Survey respondents also raised concerns about the implications of the pandemic for future HTA processes, described in around one-quarter of practical challenges. Most of these related to future COVID-19 assessments, which will need a process of continually monitoring for new evidence (such as trials of new technologies or in novel variants, new data cuts, or peer-reviewed publications of previously unpublished data), and reviewing whether new evidence affects decisions that were originally made rapidly and under substantial uncertainty. At the roundtable, this process of ‘rolling’ review was described as a shift away from usual processes, but one that may be the springboard for HTA agencies to apply it more regularly in other disease areas, shifting from technology assessment to technology management [6]. Responses also described difficulties involving stakeholders in assessments during the pandemic, with reduced engagement from technology developers and clinical experts increasingly busy with frontline care.

Political Issues

The survey identified several challenges faced by HTA agencies derived from wider, political issues. Over three-quarters of these were grouped into a subtheme of external pressures on HTA agencies; for example, a pressure from policymakers to act quickly, potentially compromising the quality of assessments, and a societal pressure to approve treatments. Four responses noted that low-quality evidence can be misinterpreted, and even cherry-picked, which can influence public expectations and place HTA agencies in a difficult position to conduct independent assessments. Roundtable participants recognised the external pressures. One noted that they are closely linked to other challenging issues in the ‘decision making’ theme, namely the usefulness of economic evaluation, and uncertainty about the appropriate threshold. Another explained that HTA agencies are familiar with their assessments being closely scrutinised by lobby groups in a specific disease area, but this has been augmented by the far-reaching effects of the COVID-19 pandemic within and beyond healthcare. Similarly, a participant advised that there is a tendency for all stakeholders to expect their technology, disease area or decision problem to be considered special in some way, which may be true for COVID-19 technologies in the immediate response to a global pandemic but not as healthcare systems return to normality over time.

Other political challenges elicited from the survey included the unusual context of having to conduct assessments against a backdrop of limited global supply of vaccines and treatments and related ethical concerns (for example, of supply being channelled to high-income countries). A roundtable attendee noted that competition between countries for limited global supply have further reduced the appetite for thorough cost-effectiveness assessments to inform decision making.

At the roundtable discussion, two additional themes emerged:

Responding to the Challenges

To keep up with the pace of new evidence, some agencies have adapted their decision-making processes by implementing rapid reviews and accepting lower-quality evidence. A roundtable attendee explained that their rapid COVID-19 assessments had initially been undertaken without the usual submission of evidence from manufacturers. This was itself challenging, and the agency has since reinstated the need for companies to submit evidence dossiers for COVID-19 technologies. Another attendee explained the rapid review format introduced by their agency, whereby companies engage with the HTA process now have the incentive of 6–12 months of temporary reimbursement, but with the commitment of having a full assessment during that time.

Another participant explained that the difficult pandemic circumstances have prompted their agency to find innovative ways of supporting decision making. For example, they have developed a dedicated evidence portal containing COVID-19 assessment reports, data and news, to rapidly disseminate relevant information to decision makers. Furthermore, the attendee noted that it had become apparent that different HTA agencies internationally, and different organisations within their own national setting, had been duplicating effort by working toward similar objectives in isolation during the pandemic. Subsequently, their response to COVID-19 has taken a more collaborative approach, and those relationships and knowledge-sharing initiatives may benefit future activities beyond the pandemic.

The Changing Role and Contribution of Health Technology Assessment

Several roundtable discussion points related to the continually evolving context of the pandemic over time, and participants reflected on the role of HTA at different stages. It was commented that in an emergency where policymakers prioritise securing access to clinically effective treatments, the remit of HTA agencies during the urgent response period may need to shift toward providing value in other areas.

First, participants suggested that agencies could seek to intervene early in the procurement process, conducting exploratory or early assessments to help policymakers understand what a fair price might be for a COVID-19 technology. It is likely that this input would be less robust than a full HTA but would still increase the probability of healthcare systems identifying a value-based price without compromising their need to act quickly. One participant suggested that static pricing arrangements are unlikely to be optimal in the context of a rapidly changing pandemic. Other models that reduce risk (for example, linking reimbursement to clinical outcomes) may be useful to mitigate uncertainty and could be linked with rolling HTA review processes.

Second, participants proposed that agencies could provide more implementation advice to support the rollout of technologies that decision makers have decided to reimburse without formal HTA. For example, agencies could evaluate the optimal sequence of treatments, or identify subgroups for whom technologies are most cost effective, to assist with prioritising treatments that are subject to limited supply. A roundtable participant whose agency was engaged in implementation-related activity explained that this provides an opportunity to ensure technologies are at least being used as efficiently as possible.

Participants also commented that, longer term, better coordination between HTA agencies and the policymakers that commission assessments would be beneficial, to provide clarity on the role of HTA ahead of the next pandemic. An attendee cited their agency’s experience of being asked to conduct the same evaluation multiple times in response to the changing evidence base. It may be useful for HTA agencies to take the initiative and suggest areas where they can add the most value to decision making in the urgent parts of a pandemic, such as the pricing and implementation functions described above.

Lastly, attendees discussed the medium-term role of HTA as societies begin to recover from the pandemic. Participants felt that decision-making regarding COVID-19 technologies is somewhere on a path back to ‘normal’ HTA, where agencies would seek to move away from urgent procurement decisions made with limited HTA involvement, and instead apply their usual assessment methods. However, many of the challenges identified here will persist for some time. All roundtable participants indicated that consensus-based methodological guidance for assessing COVID-19 technologies during the intermediate recovery phase—as a step towards the post-pandemic return to full, rigorous HTA—would be valuable to their agencies.

Discussion

The findings of our study show that HTA agencies around the world are facing similar challenges in their response to the COVID-19 pandemic. These include a lack of suitable clinical effectiveness evidence, and limited reliability, duration and generalisability of the evidence that is currently available. Agencies are more dependent on sources of evidence that have a higher risk of bias, such as observational studies and preprint articles, trading off robustness and certainty for timeliness. The weak clinical evidence base also leads to a lack of meaningful cost-effectiveness evidence. How to assess cost effectiveness, in terms of the appropriate perspective and willingness-to-pay threshold, remains challenging for HTA agencies.

Many of the issues identified are consistent with, or closely linked to, those predicted by Leahy et al. [5], who also consider likely challenges posed by the pandemic for HTA of other disease areas. For example, trial and resource use data collected during the pandemic for non-COVID-19 conditions might not be generalisable to future assessments. Trials may be subject to abnormally high discontinuation due to COVID-19 infections. Services may have moved to lower-cost telephone-based appointments or, conversely, to a higher cost-per-appointment, due to time spent disinfecting clinics between in-person attendances.

Lorgelly and Adler highlight uncertainty about the appropriate cost-effectiveness threshold during a pandemic [6], which emerged as a rich roundtable discussion point. Padula et al. also demonstrate this, reporting a uniform split of preferences between hypothetical US threshold values ranging from $50,000 to $180,000 per quality-adjusted life-year gained [7]. The same authors also suggest the high societal burden of COVID-19 implies a societal perspective should be taken. Our findings provide some support for this view among HTA agencies. However, it would require complex modelling to estimate opportunity costs across multiple sectors and should be considered carefully [14]. At the roundtable, there was some caution among participants about reacting to the pandemic in ways that might set difficult precedents for future assessments and lead to inconsistent decisions if only applied for COVID-19 technologies. Furthermore, Padula et al. suggest that a societal perspective may nullify the usefulness of economic evaluation, as an effective technology is likely to confer massive societal benefits and cost savings during a major global pandemic.

HTA agencies are accustomed to assessing technologies with a lack of high-quality evidence, such as for rare diseases and medical devices, but the speed with which new COVID-19 evidence is becoming available does appear to be a novel challenge that HTA agencies need to adapt their methods to accommodate. The use of a ‘living’ guideline approach has gained prominence during the pandemic, with some clinical guideline developers such as NICE and WHO adopting this approach for their COVID-19 treatment guidelines as a way of responsively reacting to new information [15, 16]. This approach could similarly be explored by HTA agencies. A living approach to COVID-19 assessments would align with previous proposals to implement a lifecycle approach in HTA [17, 18] and transition to health technology management [4].

In addition to the speed of evidence generation, HTA agencies will also need to conduct their assessments of COVID-19 technologies under time pressure. During the intermediate pandemic recovery phase, many novel or repurposed technologies for COVID-19 will require a reimbursement decision. Furthermore, healthcare systems will have a range of COVID-19 technologies in clinical use that were not subjected to full HTA and were perhaps funded by external budget increases. Payers are likely to seek value for money from those treatments [6]. To respond quickly to the challenges identified here, a novel framework for assessing the relative value of COVID-19 technologies may be needed [7]. Such interim methods and processes should inform future pandemic preparedness within healthcare systems, in the same way that lessons were learned from the 2009 influenza pandemic [19].

Our findings are consistent with those of Padula et al., who posit that the pandemic has demonstrated how closely intertwined healthcare decisions and outcomes are with wider socioeconomic decisions and outcomes [7]. They are also in line with the general conclusions from O’Rourke et al. [20], who flagged the conundrum facing HTA agencies in balancing rigor and speed as one of the key challenges. However, our study offers richer and more recent insights, given its triangulation of data from survey responses and the HTA roundtable workshop.

When asked whether there is a need for methodological and process guidance to support HTA agencies during the current intermediate phase of the pandemic, the response from the HTA roundtable attendees was overwhelmingly positive. The findings of our study will thus be used to inform the development of a best-practice guidance for the assessment of COVID-19 technologies, with special focus on diagnostics and therapeutics, which are the most assessed interventions by HTA agencies. This guidance is currently being developed within the HTx project and its newly established ‘policy sandbox’ [21]. A policy sandbox is a safe space to promote innovation and test novel ideas. This will be the first use of the sandbox concept in HTA, to co-develop the guidance with various HTA stakeholders, including methodologists, payers, clinicians, industry, and patient representatives.

Limitations

In our thematic review, several of our key themes may appear to overlap; for example, decision-making challenges are influenced by both external pressures and challenges in assessing cost effectiveness.

There are some limitations to the generalisability of our findings. First, each HTA agency was asked to provide a single response to our survey, which might not be wholly representative of the agency’s experience. However, we provided a Microsoft Word version of the survey to facilitate recipients in collaborating with colleagues within their agencies, and received evidence that many respondents were doing so. Second, we conducted our survey and roundtable 1 year after the initial peak of the pandemic. Over time, the challenges faced by HTA agencies may evolve, with early issues such as the lack of clinical evidence becoming less acute, replaced by new key issues. Additionally, we attempted to maximise the geographical generalisability of the findings by distributing the survey to many global HTA agencies, but were not able to reach any agencies in Africa or South America, and only one in Asia. These regions are therefore underrepresented in the findings. However, we believe that supplementing our survey with the in-depth roundtable discussion with agencies from both high- and middle-income countries allowed us to collect comprehensive information, and data saturation is likely to have been reached. This supports the applicability of our findings to other geographical regions.

Finally, we focused on the challenges faced and perceived by HTA agencies. Decision makers, such as healthcare payers, may have different challenges and expectations of HTA agencies in their assessments of COVID-19 technologies. This would be informative future research.

Conclusions

To our knowledge, this is the first collaborative effort between many global HTA agencies to share knowledge about challenges relating to COVID-19 assessments and how to face and learn from them. Key issues include a lack of high-quality clinical effectiveness evidence, with high uncertainty and heterogeneity; difficulties in estimating and assessing cost effectiveness; external pressures to act quickly and approve treatments; and the practicalities of rapid working and capacity constraints. Many of the challenges identified will remain problematic during the pandemic recovery period. It appears that an interim framework of good-practice guidance for HTA agencies, providing a common set of consistent, pragmatic approaches to address the key challenges identified here, would be valuable during this intermediate phase. Such guidance would help agencies step up the rigor of COVID-19 technology assessments as healthcare systems continue on their paths back to normality.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 42 KB)

There are substantial barriers to the health technology assessment (HTA) of diagnostic and therapeutic technologies for coronavirus disease 2019 (COVID-19).

HTA agencies are familiar with some of the methodological challenges, such as the paucity of high-quality evidence; however, others are more specific to the pandemic context, such as the logistical challenges of rapidly evolving scientific understanding of COVID-19 and clinical practice, and external pressures on HTA agencies to act quickly and approve effective technologies.

HTA agencies would welcome interim guidance to begin increasing the robustness of their assessments of these technologies.