How aligned are the perspectives of EU regulators and HTA bodies? A comparative analysis of regulatory-HTA parallel scientific advice.

BACKGROUND: In 2010, the European Medicines Agency (EMA) initiated a pilot project on parallel scientific advice with Health Technology Assessment bodies (HTABs) that allows manufacturers to receive simultaneous feedback from both the European Union (EU) regulators and HTABs on their development plans for medicines. AIMS: The present retrospective qualitative analysis aimed to explore how the parallel scientific advice system is working and levels of commonality between the EU regulators and HTABs, and among HTABs, when applicants obtain parallel scientific advice from both a regulatory and an HTA perspective.
METHODS: We analysed the minutes of discussion meetings held at the EMA between 2010, when parallel advice was launched, and 1 May 2015, when the cutoff date for data extraction was set. The analysis was based on predefined criteria and conducted at two different levels of comparison: the answers of the HTABs vs. those of the regulators, and between the answers of the participating HTA agencies.
RESULTS: The analysis was based on 31 procedures of parallel scientific advice. The level of full agreements was highest for questions on patient population (77%), while disagreements reached a peak for questions on the study comparator (30%). With regard to comparisons among HTABs, there was a high level of agreement for all domains.
CONCLUSIONS: There is evident commonality, in terms of evidence requirements between the EU regulators and participating HTABs, as well as among HTABs, on most aspects of clinical development. Indeed, regardless of the question content, the analysis showed that a high level of overall agreement was reached through the process of parallel scientific advice.

What is Already Known about this Subject

Designing a study suitable for regulatory approval might not always translate into data suitable for reimbursement decisions.

Health Technology Assessment bodies (HTABs) operate under different methodological and national legal frameworks.

Parallel scientific advice allows manufacturers to receive simultaneous feedback from both the European Union regulators and HTABs on their development plans for new medicines.

What this Study Adds

There is evident commonality, in terms of evidence requirements between the EU regulators and HTABs, as well as among HTABs, on several aspects of clinical development.

Regardless of the question content, a high level of overall agreement is reached through the process of parallel scientific advice.

There is no evidence of blurring of remits between the EU regulators and the HTABs.

Background and objectives

Achieving patient access for a new medicinal product typically involves several steps. Approval by a regulatory body to obtain a marketing authorization is based on the benefit/risk assessment, a process that requires the evaluation of quality, nonclinical and clinical data submitted by the applicant, excluding any economic considerations. In the European Union (EU), the establishment of the Community authorization has led to procedures for drug approval on a European‐wide basis for many medicinal products. Following regulatory approval, subsequent decisions on the coverage (reimbursement) and price of an authorized drug are made at the national level in each EU Member State. In countries where Health Technology Assessment (HTA) is in place, third‐party payers, pricing and reimbursement agencies/HTA bodies (HTABs) rely upon HTA mainly to determine the reimbursement status of a drug and to support the price negotiation process 1. The role of payers is to optimize the health outcomes for the population by considering all available treatment options while accounting for budgetary constraints 2.

Compared with regulatory bodies – focusing on the benefit/risk assessment of a product, typically evaluated in the rigorously controlled setting of randomized controlled trials, with close attention to internal validity, safety, efficacy and manufacturing – HTABs have different remits and, therefore, additional evidence requirements. Criteria for reimbursement decisions vary across countries but can include unmet medical needs, the relative effectiveness and safety of the drug, drug price, budget impact and cost‐effectiveness 3. Designing a study suitable for regulatory approval might not always translate into data suitable for reimbursement decisions, creating an evidence gap between the regulatory and reimbursement submission and posing a hurdle to patient access to new drugs in some countries 4. Furthermore, HTABs operate under different methodological and national legal frameworks, often resulting in divergent outcomes 5. Indeed, even when a marketing authorization is obtained, HTABs and decisions on drug pricing and reimbursement often delay access to medicines for patients at a national or even regional level 6.

In 2010, the European Medicines Agency (EMA), together with HTABs, established a pilot project for parallel scientific advice that allows manufacturers to receive simultaneous feedback from both the EU regulators and HTABs on their development plans for new medicines 7. The aim of parallel scientific advice is to ensure that the relevant evidence is collected for each stakeholder in an efficient way. A single advice is provided by EU regulators in this process, consolidated at the EU level, with the EMA coordinating the resources put at its disposal by the EU regulatory medicines network for this purpose. Participation of HTABs currently depends on the applicant's request and varies based on the availability of HTABs. Each HTAB generally provides its own institutional advice through the face‐to‐face meeting with the applicant and the regulators, which is reported in minutes, and now more recently also in written reports by some HTABs.

The objective of the analysis reported here was to explore how the parallel scientific advice system is working and the level of commonality between the EU regulators and participating HTABs, and among HTABs themselves, when applicants obtain simultaneous scientific advice from both a regulatory and an HTA perspective. The analysis focused on the outcome of the discussion meeting with all stakeholders – the applicant, HTABs and the regulators – which is the final procedural step of parallel scientific advice.

Methods

Data collection

The present comparative analysis was based on the minutes of discussion meetings held at the EMA between 2010, when parallel advice was first launched, and 1 May 2015, when the cutoff date for data extraction was set. The minutes were used to extract and compare the answers provided by the regulators and HTAB representatives to each question posed by the applicant in the submission. Each submission consisted of a variable number of questions asked by applicants to both the regulators and HTABs or exclusively to one of the two. Questions addressed exclusively to the regulators, not allowing the comparisons between the regulators and HTABs, were excluded from the analysis. Similarly, questions addressed only to HTABs were excluded from the comparisons between the regulators and HTABs, and used for the comparisons among HTABs only.

A standardized form was designed to list and describe all questions for each procedure as well as the applicant's position and the scientific advice provided by both the regulators and HTABs in response to each question. The form was initially tested on six procedures, chosen randomly, and then agreed by the research team. Each question, being the item of analysis, was grouped separately under domains and subdomains based on its content (see Table 1). When a single question contained ‘subquestions’ with different contents, these subquestions each became an item for analysis and were therefore classified in domains and subdomains.

Table 1

Domains and subdomains used to classify question content

DOMAINS	SUBDOMAINS
Population	• Inclusion and exclusion criteria
	• Therapeutic indication
	• Biomarkers and subgroups
	• Extrapolations
Comparator
Endpoints	• Primary efficacy endpoint
	• PROs and HRQL
	• Secondary endpoints (not including PROs)
	• Clinical relevance of the effect size
Other study design characteristics	• Randomization
	• Treatment duration
	• Statistical analysis methods
	• Dosing
Overall efficacy and safety data package	• Strategic questions
	• Safety database
Economic evaluation (only for HTABs)	• Economic model
	• Data for economic analysis
	• Indirect comparisons

The ‘strategic questions’ were questions in which general feedback about the clinical efficacy programme was sought for registration and/or pricing/reimbursement purposes. HTABs: Health Technology Assessment bodies; HRQL: Health‐Related Quality of Life; PRO: Patient‐Reported Outcomes.

Subsequently, the comparison of the answers provided by the regulators and HTABs was performed in a blind fashion, by two appraisers for each question, in order to identify the level of agreement. The analysis of the level of agreement was assessed independently, based on predefined criteria (see Tables 2 and 3), by both of the appraisers, masked to the outcome of the other. Following the independent assessment of the level of agreement, results were then cross‐checked, leading to a joint document. In the case of disagreement between the two appraisers, the final decision was made through a consensus process reached following a discussion based on the predefined criteria and on previous similar cases analysed, in order to ensure consistency.

Table 2

Criteria to define agreement between the regulators and Health Technology Assessment bodies (HTABs)

Agreement	The HTAB clearly expressed full agreement with the regulators' answer
Partial agreement	The HTAB expressed general alignment with the regulators, although raising minor concerns or adding minor requirements not mentioned by regulators
Disagreement	The HTAB expressed a clear disagreement with the regulators. Alternatively, the HTAB raised major concerns or added major requirements not mentioned by the regulators
Not assessable	No answer or no clear answer was reported in the minutes, thus hampering the comparison with the regulators

Table 3

Criteria to define agreement among Health Technology Assessment bodies (HTABs)

Agreement	The answers provided by the two HTABs were fully aligned. Alternatively, overall alignment was found between the two, although one raised minor additional concerns or added minor additional requirements
Disagreement	There was complete disagreement between the two HTABs. Alternatively, one raised major concerns or added major requirements not mentioned by the other
Not assessable	No answer or no clear answer was reported in the minutes, thus hampering the comparison between the two

The analysis of the level of agreement was conducted at two different levels of comparison:

The answers of the HTAB vs. those of the regulators: for each question, the answer of each HTAB was compared with that of the regulators, which was used as a reference.

HTAB vs. HTAB answers: for each question, the answer of each HTAB was compared with those of all other HTABs in a pairwise fashion, resulting in multiple comparisons. The maximum number of multiple comparisons was k*(k–1) /2, where k represents the number of HTABs involved in a specific parallel advice procedure.

The comparisons between the answers of the HTABs and those of the regulators were based on three categories – i.e. ‘full agreement’, ‘partial agreement’ and ‘disagreement’ (see Table 2). A modified set of criteria was used for the comparisons between HTABs based on two categories – i.e. ‘agreement’ and ‘disagreement’ (see Table 3). Indeed, to facilitate the assessment of agreement in the multiple HTAB vs. HTAB comparisons, and to deal with the complexity of basing such assessments exclusively on meeting minutes, a pragmatic approach, based on this modified set of criteria, was adopted by the research team.

When a particular question was not accompanied by a clear answer in the minutes, no level of agreement could be assessed for that HTAB, and such cases were classified as ‘not assessable’ (see Tables 2 and 3 for details).

A series of examples of how the comparisons were carried out and classified is listed in Tables S1 and S2.

Data analysis

Descriptive statistics (absolute values and percentages) were used to describe the level of agreement between HTABs and the regulators and among HTABs.

In comparing the answers of HTABs against those of the regulators – considering the varying number of participating HTABs in the analysed procedures – the frequency (%) of different levels of agreement for each domain was calculated, summing the total number for that category (i.e. either full agreements, partial agreements or disagreements), for each domain, as a percentage of the total number of HTABs answering those questions in that domain. For example, in order to calculate the frequency of disagreements within the comparator domain, we summed the number of HTABs expressing a disagreement on all the questions related to the comparator across all procedures. This number was then divided by the overall number of HTABs expressing an opinion (either a full/partial agreement or disagreement) on the same questions (examples of calculation are provided in Table S3).

While the category of ‘not assessable’ was excluded from the analysis, a sensitivity analysis was performed in order to estimate the potential bias stemming from its exclusion. Therefore, ‘not assessable’ answers were imputed as full agreement or disagreement for each domain.

In comparing HTAB against HTAB answers, the same approach was used to calculate the frequency (%) of the agreements and disagreements within each domain, again excluding the ‘not assessable’. In this case, the percentage was based on the multiple comparisons among HTABs. For example, in order to calculate the frequency of disagreements within the comparator domain, we calculated the total number of multiple comparisons assessed as disagreements on all the questions related to the comparator across all procedures. This number was then divided by the overall number of multiple comparisons among HTABs assessed as either an agreement or a disagreement on the same questions (examples of calculation are provided in Table S4).

All cases of disagreement on the comparator were examined in further detail to see whether such disagreements between the regulators and HTABs could potentially be addressed within a single development plan. In addition, to examine for ‘contamination’ or blurring of remits between the regulators and HTABs, a random sample (15%) of cases of full or partial agreement between the regulators and HTABs on the comparator was examined in detail by two appraisers, assessing whether or not there were additional reasons underpinning the choice of comparator for the regulators beyond furthering the understanding of safety or efficacy, or the draft regulatory ‘Reflection paper on the need for active control in therapeutic areas where use of placebo is deemed ethical and one or more established medicines are available’ 8.

Results

Overall, 43 procedures were selected initially. Six were excluded as their minutes were drafted in a general way, not allowing comparisons; three procedures were excluded as they represented follow‐up procedures of previous parallel advice with no involvement of HTABs and no discussion meeting; and three additional procedures were excluded because they were conducted within the ‘Shaping European Early Dialogues’ (SEED) Consortium 9. Consequently, 31 procedures were included in the comparative analysis between the regulators and HTABs. One of the selected procedures involved only a single HTAB and therefore could only be used for the comparisons of the HTAB against the regulators. Overall, the median number of HTABs involved per procedure was three, ranging between one and five. The main therapeutic categories represented in the parallel advice requests were: (i) oncology/immunology (13 out of 31; i.e. 42%); (ii) central nervous system (six out of 31; i.e. 19%); (iii) respiratory system (four out of 31; i.e. 13%). The remaining procedures were related to the cardiovascular system (two out of 31, i.e. 6%), infectious diseases (two out of 31, i.e. 6%) and drugs for blood or blood forming organs (two out of 31, i.e. 6%), the alimentary tract and metabolism (one out of 31, i.e. 3%), and the musculoskeletal system (one out of 31, i.e. 3%). Out of the overall requests, 45% (14 out of 31) were related to biological/biotechnological products, 45% (14 out of 31) were related to small molecules and three to advanced therapies. In three cases (three out of 31; i.e. 10%), companies requesting parallel advice were small and medium‐sized enterprises (SMEs). In four cases out of 31, the request was related to an orphan drug.

Eight different HTABs participated in parallel scientific advice (see Figure 1 for a complete list of participating HTABs and participation rates). The most frequently represented HTAB was the National Institute for Health and Care Excellence (NICE; involved in 90% of all parallel advice procedures), followed by the German Federal Joint Committee (G‐BA; 65%), Italian Medicines Agency (AIFA; 45 %), Dental and Pharmaceutical Benefits Agency (TLV; 35%), National Authority for Health (France) (HAS; 19 %), Main Association of Austrian Social Security Institutions (HVB; 10%), Catalan Agency for Health Quality and Assessment (AQuAS; 10%), and National Institute for Sickness and Invalidity Insurance (INAMI; 3%).

Figure 1

Participation of Health Technology Assessment bodies (n = 31). AIFA, Italian Medicines Agency; AQuAS, Catalan Agency for Health Quality and Assessment; G‐BA, German Federal Joint Committee; HAS, National Authority for Health (France); HVB, Main Association of Austrian Social Security Institutions; NICE, National Institute for Health and Care Excellence (England); INAMI, National Institute for Sickness and Invalidity Insurance (Belgium); TLV, Dental and Pharmaceutical Benefits Agency (Sweden)

Overall, 375 questions and 588 answers from HTABs were analysed to assess the level of agreement between HTABs and the regulators. Out of all the answers analysed, 317 were assessed as full agreements, 118 as partial agreements, 83 as disagreements and 70 as ‘not assessable’ (see Figure 2). Excluding the ‘not assessable’ answers (i.e. 588–70 = 518), full agreements reached 61% (317/518), partial agreement 23% (118/518) and disagreements 16% (83/518).

Figure 2

Absolute values based on the total number of health technology assessment body (HTAB) opinions provided across 31 procedures (comparison regulators vs. HTABs)

The level of full agreement between HTABs and the regulators was high across all domains (see Figure 3 and Table S5), at 77% for answers on patient population, 60% each for questions related to endpoints and to other study design characteristics (such as treatment duration, statistical analysis, dosing and randomization), 59% for the overall efficacy and safety data package, and 44% for the study comparator. The level of disagreement was highest on questions related to the comparator (30%), and was 23% for questions about the overall efficacy and safety data package, 21% for other study design characteristics, 12% for study endpoints and 9% for patient population.

Figure 3

Level of agreement for each domain: Health Technology Assessment bodies (HTABs) vs. regulators (based on 31 procedures). n represents the total number of HTABs expressing an opinion for each domain. full agreement partial agreement disagreement

In order to assess the potential bias resulting from the exclusion of ‘not assessable’ answers from the analysis, a sensitivity analysis was performed for each domain (see Table S6). It showed that imputing ‘not assessable’ as either ‘full agreements’ or as ‘disagreements’ did not influence the interpretation of data and the pattern of agreement (see Table S7).

Regarding the level of agreement among HTABs, 364 questions were used for 713 multiple comparisons, 568 of which were assessed as ‘agreements’, 82 as ‘disagreements’ and 63 as ‘not assessable’. In this case, agreements comprised 87% of the total (568/650), excluding the ‘not assessable’ answers (713–63 = 650).

With regard to the comparisons among the HTABs for each domain, study findings showed a high level of agreement for all of them, ranging from 71% in the domain of other study design characteristics to 94% for questions on the population (see Figure 4 and Table S8). The level of disagreement was 29% for questions on other study design characteristics, 21% for questions on the comparator, 13% for those related to the overall efficacy and safety data package, 12% for economic evaluation and 6% for the patient population.

Figure 4

Level of agreement among Health Technology Assessment bodies (HTABs) for each domain (based on 30 procedures). n represents the number of multiple comparisons among HTABs expressing an opinion for each domain. agreement disagreement

Further analyses showed the composition of the cases of disagreements within each domain and subdomain (see Figures S1 and S2).

A subanalysis focusing on the procedures in which there was one or more HTAB in disagreement with the regulators on the comparator was performed. Seven such cases were identified and examined in further detail to see whether such disagreements could have been potentially addressed within a single trial, a single development plan or other approaches. In four cases, an indirect comparison was discussed as a potential solution. Caveats with such an approach surrounded the need for justification of the indirect approach and the impact on increasing uncertainty. In the other three cases: (i) a redesigned trial was needed to address the issues related to the comparator but also other dimensions of the trial; (ii) in two cases, a three‐arm trial was the most obvious solution, potentially acceptable to both the regulators and the HTABs.

In addition, to test for ‘contamination’ or blurring of remits between the regulators and HTABs, a random sample (15%) of cases of full‐ or partial agreement on the comparator between the regulators and HTABs was examined in detail, assessing whether or not there were ‘non‐regulatory reasons’ behind the choice of comparator for the regulators. Based on this sample, there was no evidence of blurring of remits, and the choice of the comparator for the regulators was firmly anchored in an expected regulatory rationale.

Discussion

The present article represents the first systematic attempt to look into the level of agreement reached by the EU regulators and participating HTABs for the recommendations provided when giving parallel scientific advice. This work is a contribution to the international discussion and debate on models of increased engagement and cooperation between technology developers, regulators and payers/HTABs. It is recognized that regulators and HTABs have different objectives, so a variation in data requirements may be expected. The objectives of parallel scientific advice are to ensure that evidence to meet the needs of the respective decision makers are collected in an efficient and optimum way and that, ultimately, major objections at the marketing authorization application stage and at health technology appraisal, stemming from data collection, are avoided. High levels of dialogue between different stakeholders at an early stage help to increase the knowledge and understanding of the possible differences and perspectives of different stakeholders. In addition, through discussion and, where possible, alignment of views and requirements, the data will be more likely to meet the needs of the stakeholders in a feasible and efficient manner while still respecting the roles and remits of the various bodies. Methods to address the identified divergences are needed when differences in data requirements cannot be resolved.

All of the analysed procedures were conducted under the draft best practice guidance 10. A public consultation on this indicated a high level of support for this activity and included constructive suggestions for change.

One of the key findings of this analysis was that, regardless of the question content, a high level of overall agreement between the regulators and HTABs was evident at the culmination of the process. It is important to stress that clarifications provided by the applicant and exploration of the possible other ways forward (see examples in Table S9) were instrumental in reaching this final level of agreement, minimizing divergences and identifying possible solutions as described in the minutes of the face‐to‐face meeting. For example, in one of the procedures considered for the present analysis, the regulators agreed with the use of placebo as a study comparator. By contrast, the HTABs requested the use of an active comparator. The choice of placebo as comparator was discussed extensively at the meeting with the applicant. The resulting strategy consisted of treating patients with standard of care using the experimental drug on top of the active arm, and was finally agreed by all parties (see Table S9). Another example was a discussion on the primary endpoint, within a different procedure. In such cases, both the regulators and HTABs agreed that the proposed surrogate endpoint was acceptable overall but some of the HTAB representatives indicated a need to show correlation of the surrogate endpoint with clinical outcomes and quality of life. During the discussion meeting, the applicant proposed a new composite key secondary endpoint in support of the primary endpoint, as a means to accommodate patient heterogeneity and quality of life, and this was considered acceptable by HTABs (see Table S9). These examples show that early dialogue can be a worthwhile process for all parties and can lead to common understanding about evidence development for market access. Alternatively, this process can represent the opportunity for companies to receive a clear ‘red light’ message on certain aspects of drug development. Therefore, this exercise can guide applicants to invest resources in viable developments from both a regulatory and a reimbursement perspective, to provide the required evidence to support regulatory and reimbursement decision making, and to have a timely access to the market in the interest of patients.

The analysis also showed that HTAB views among the subset of participating HTABs are not as fragmented as might have been considered, given the differing HTA appraisals and reimbursement decisions across EU Member States 11, 12, 13. It also confirmed the findings of a recent study produced by the European Parliament, exploring how relative effectiveness assessments are conducted in different Member States, which discovered that the underlying principles are not fundamentally incompatible and share the same goals and concepts 14. Broader HTAB participation in parallel advice with regulators would further enhance the representativeness and value of the interaction.

Our findings can be seen in the context of stronger interactions between regulators and HTABs, and knowledge sharing among HTABs promoted at the European level through various initiatives over the last few years 15, 16, 17. European collaboration on HTA has, indeed, been recognized as a strategic priority, in which stakeholders have been investing substantial resources 18. In addition, the EMA's recently launched adaptive pathways for bringing new medicines to the market have extended the collaboration between regulators, companies, HTABs and payers beyond the concept of parallel scientific advice, throughout the entire product life span 19.

Of note, only one of the procedures included in this analysis is associated with a European marketing authorisation, while another one is related to a submission currently being reviewed by the Committee for Medicinal Products for Human Use (CHMP). This implies that assessing the impact of parallel scientific advice on the reimbursement decision‐making process will only be possible in the future once products receive marketing authorization and actually enter the market. Further research is clearly warranted in this respect.

A limitation of the analysis was that it was based on the discussion minutes prepared by the applicants, which could reflect their interpretation and understanding of the discussion. In some cases, the discussion minutes included ‘not assessable’ answers. This confirms that the supply of written advice by HTABs will better record their views, contributing to improving the overall process. It is noteworthy that a number of HTA agencies now offer a written report to the company as an outcome of the parallel process. In addition, the exclusion of the ‘not assessable’ answers may have interfered with the interpretation of the final results; however, following a sensitivity analysis, the impact of the latter answers on the overall pattern of agreement was considered minor. Another potential limitation was the subjectivity of the assessment of the agreements. However, this subjectivity would not have threatened the validity of the study, as predefined assessment criteria were used and the assessment was conducted in a blinded fashion.

It is recognized that alignment on advice is not reached in all cases, considering that the remits and requirements are different for each decision maker. The essential question is whether the remaining evidentiary requirements can be accommodated in a single development plan, single trial or other methods. There is a potential need to address the remaining divergences if these are deemed critical and represent difficult development plan tradeoffs in terms of which evidence point to address 18. Such critical divergences could be further discussed through EMA parallel regulatory HTA follow‐up procedures, qualification procedures, broad advice or workshops 20. The possibility of having to use indirect comparisons is recognized, but inherent in this approach are potential methodological difficulties and increased uncertainty. There have been calls for joint regulatory HTA disease‐specific guidelines to address such areas. However, guidelines require a substantial body of experience within a therapeutic area and to be kept up to date, and may still not cover all particular cases. As such, individual parallel advice procedures can be targeted to particular development programmes, and, ultimately, all learning can be distilled and, where needed, guidelines drafted.

In conclusion, the present retrospective analysis of parallel scientific advice procedures showed that there is evident commonality, in terms of evidence requirements, between the EU regulators and participating HTABs, as well as among HTABs, on several aspects of clinical development. Ideally, these findings should be confirmed by further prospective research based on a larger sample of procedures.

Disclaimer

The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of, or reflecting the position of, any national competent authority, the EMA, or one of its committees or working parties. These comments should not be used as ‘scientific guidance’ and are not intended to fulfil this function on specific products.

Competing Interests

All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

Table S1 Selected examples showing how the comparisons between regulators and HTA Bodies (HTABs) were carried out and classified. Confidential information was omitted/anonymised

Table S2 Selected examples showing how the comparisons amongst HTABs were carried out and classified. Confidential information was omitted/anonymised

Table S3 Example of calculation of the level of agreement/partial agreement/disagreement between regulators and HTABs

Table S4 Example of calculation of the level of agreement/disagreement among HTABs

Table S5 Levels of agreement, partial agreement and disagreement between regulators and HTABs per question content (domains and subdomains)

Table S6 Sensitivity analyses for the “Not assessable” category (absolute values)

Table S7 Findings based on the exclusion and inclusion of not assessable answer

Table S8 Levels of agreement and disagreement amongst HTABs per question content (domains and subdomains)

Table S9 Examples of discussion at face to face meetings. Confidential information was omitted/anonymised

Figure S1 Compositions of the disagreement cases between regulators and HTABs

Figure S2 Compositions of the disagreement cases among HTABs

Supporting info item

Click here for additional data file.

Supporting info item

Click here for additional data file.

Supporting info item

Click here for additional data file.