Can the EVIDEM Framework Tackle Issues Raised by Evaluating Treatments for Rare Diseases: Analysis of Issues and Policies, and Context-Specific Adaptation.

BACKGROUND: The multiplicity of issues, including uncertainty and ethical dilemmas, and policies involved in appraising interventions for rare diseases suggests that multicriteria decision analysis (MCDA) based on a holistic definition of value is uniquely suited for this purpose. The objective of this study was to analyze and further develop a comprehensive MCDA framework (EVIDEM) to address rare disease issues and policies, while maintaining its applicability across disease areas.
METHODS: Specific issues and policies for rare diseases were identified through literature review. Ethical and methodological foundations of the EVIDEM framework v3.0 were systematically analyzed from the perspective of these issues, and policies and modifications of the framework were performed accordingly to ensure their integration.
RESULTS: Analysis showed that the framework integrates ethical dilemmas and issues inherent to appraising interventions for rare diseases but required further integration of specific aspects. Modification thus included the addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes, and economic consequences of interventions for rare diseases. Scoring scales were further developed to include negative scales for all comparative criteria. A methodology was established to incorporate context-specific population priorities and policies, such as those for rare diseases, into the quantitative part of the framework. This design allows making more explicit trade-offs between competing ethical positions of fairness (prioritization of those who are worst off), the goal of benefiting as many people as possible, the imperative to help, and wise use of knowledge and resources. It also allows addressing variability in institutional policies regarding prioritization of specific disease areas, in addition to existing uncertainty analysis available from EVIDEM.
CONCLUSION: The adapted framework measures value in its widest sense, while being responsive to rare disease issues and policies. It provides an operationalizable platform to integrate values, competing ethical dilemmas, and uncertainty in appraising healthcare interventions.

Key Points for Decision Makers

Introduction

Appraising the value of interventions for rare diseases raises a number of difficult issues. Many of these conditions are severe, chronic, progressive, and life-threatening, with multiple medical, psychological, and social consequences [1, 2]. Their rareness is a major barrier to developing targeted therapeutic interventions due to both economic and scientific constraints. As a consequence, unmet needs remain significant and widespread [3]. Uncertainty is also a critical issue in rare diseases, and includes limitations in our understanding of natural history [4] as well as challenges in obtaining high-quality epidemiological [5-8], clinical [9, 10], and economic data. In addition to uncertainty on evidence, there is variability in country or institution policies and across stakeholders on the values at stake. Regulatory policies to support the development of interventions for rare diseases are in place [11, 12] but, once approved, appraising these often highly-priced products for reimbursement is a challenge for health technology assessment (HTA) [13-15]. Some agencies acknowledge that their standard appraisal approaches need to be modified for orphan products [13], and a few have developed explicit approaches, including the National Institute for Health and Care Excellence (NICE) Interim Process for Highly Specialized Technologies (HST) [16] and the Ontario Public Drugs Program framework [17].

The multiplicity of issues involved in appraising interventions for rare diseases, as highlighted in the NICE interim HST process [16], suggests that multicriteria decision analysis (MCDA)-based approaches that apply a holistic definition of value are uniquely suited for this purpose. MCDA makes explicit which criteria are used and how they are balanced in decision making. Several multicriteria approaches designed for appraising interventions for rare diseases have been proposed, suggesting different sets of decision criteria that were selected based on various approaches [18-20]. To be justifiable, decisions regarding healthcare interventions, and thus the criteria included in a multicriteria framework, must be based on reasons that reflect society’s ethical principles and substantive values, concepts such as fairness and efficiency [21, 22], while the decision-making process must reflect procedural values such as transparency, accountability, and participation [21] to help legitimize decisions by upholding procedural justice [21, 23, 24]. Thus, there is a need for a comprehensive appraisal framework that features decision criteria that are selected based on ethical principles (i.e. social substantive values) and reflects social procedural values.

EVIDEM is an open-source, collaboratively-developed [25-30] MCDA framework designed to appraise the holistic value of healthcare interventions [31]. It has been tested [26, 27, 32, 33] and implemented [34, 35] in various real-world decision-making settings. The EVIDEM framework v3.0 is structured around the key objectives (domains) that govern healthcare decision making and define the holistic value of interventions, i.e. to optimize the health of patients, populations, and healthcare systems. Attainment of these objectives is assessed using operationalizable criteria, each of which is rooted in established ethical positions [36, 37], including deontology (imperative to help, beneficence, nonmaleficence) [38-41], utilitarianism (greatest good for the greatest number) [42, 43], distributive justice and fairness (prioritize those who are worst off) [43, 44], and virtue ethics and practical wisdom [45]. The framework was designed to meet MCDA design principles, i.e. each criterion must make a unique contribution to the value of an intervention, while the criteria set needs to be complete, operationalizable, and free of redundancies and mutual dependencies [46]. Objectives are classified into normative and feasibility domains. In the quantitative part of the framework, criteria are weighted and scored to obtain a numerical measure of the value of an intervention (Value Estimate). The qualitative part captures the impact on value of those criteria that are difficult to quantify. Key procedural values underpinning EVIDEM are transparency, pragmatism, and participatory decision making [25-27].

Because of its holistic definition of value, explicit rootedness in ethical principles and real-world application, the EVIDEM framework was selected for this study to be analyzed and further developed into a comprehensive MCDA approach that addresses rare disease issues and policies, while maintaining its applicability across disease areas.

Methodology

Identification of Specific Issues Raised by Rare Diseases

A systematic literature review was carried out to identify issues in rare diseases that potentially impact on value appraisal as well as relevant current or proposed regulatory and reimbursement policies. Full-text publications related to disease impact (including size of population), context of interventions, clinical outcomes, costs, ethical issues, and licensing and reimbursement policies were included. Case reports and articles of clinical focus describing the natural history, clinical presentation, or treatment of specific rare diseases were excluded. The principal sources of information were peer-reviewed publications identified from the PubMed/MEDLINE databases (published from January 2003 to September 2013). Using keywords such as ‘orphan disease’ and ‘rare disorder, the PubMed/MEDLINE searches yielded 6525 citations, 115 of which were reviewed in full text, and 41 [2–10, 17–20, 47–74] included in the analysis, including five publications describing current or proposed appraisal frameworks for rare diseases [17–20, 68]. Additionally, websites of major regulatory and HTA agencies, rare disease organizations and networks (e.g. ORPHANET, EUROPLAN, Office for Rare Disease Research), and bibliographies of key publications were searched for relevant information.

The selected publications were analyzed to identify issues pertaining to value appraisal. The relevant issues were classified by the decision criterion to which they pertain, using the criteria definitions and the structure of the EVIDEM framework as an analytical tool.

Analysis and Adaptation of the EVIDEM Framework from the Perspective of Issues Raised by Rare Diseases

Ethical and methodological foundations of the EVIDEM framework v3.0 [36, 37] were analyzed on a criterion-by-criterion basis to assess their relevance and implications from the perspective of issues raised by rare diseases. This analysis was used to determine whether a modification of the framework was required to enhance its usefulness for appraising interventions for rare diseases, while preserving its applicability across disease areas. Potential modifications included:

definition of subcriteria to allow a more differentiated assessment of specific criteria;

further development of scoring scales;

structural transformation of the framework by moving criteria that are assessed qualitatively into the quantitative part of the framework to integrate context-specific policies and priorities.

Results

The adapted EVIDEM framework is shown in Fig. 1 (see Online Resource 1 for the full framework). For the quantitative part of the framework (a), hierarchical point allocation was selected as the primary weight elicitation technique, a method that allows direct expression of personal values and makes explicit the trade-offs that need to be made across the criteria [75]. As in the original framework, weighting is performed independently of the intervention appraised (generic weighting), with the exception of the subcriteria defining intervention outcomes, for which disease-specific weights are elicited to define the relative weight of each outcome as part of the Comparative Effectiveness criterion. Evaluators score the intervention with respect to each criterion using constructed, cardinal scoring scales, designed to measure their judgments on the evidence presented. These scales include scores of zero, corresponding to situations where the intervention has no value with respect to a criterion. The framework was adapted to include negative scores for all comparative criteria to reflect worse outcomes or economic consequences than comparators, which are relevant to rare diseases, but also in general for fair appraisal of interventions. The impacts of contextual criteria on the value of the intervention are considered qualitatively using a separate tool (b).

Fig. 1

Adapted EVIDEM MCDA framework: (a) quantitative MCDA model (with hierarchical structure and weighting method); and (b) contextual criteria for qualitative appraisal. MCDA multicriteria decision analysis; PRO patient-reported outcome

Criteria-specific analyses and adaptations are summarized in Table 1 and are reported below.

Table 1

Adapted MCDA framework structured by objective with ethical foundations, value contributions, rare disease issues, and adaptations made for each criterion

Objectives/criteria	Ethical foundation	Value contribution	Rare disease issues	Adaptation/comments
Normative universal objectives: quantitative appraisal
Objective: addressing areas of high therapeutic need
Disease severity	Alleviate suffering in those who are worst off (theory of justice)	Interventions targeting (i.e. preventing, curing, or alleviating) severe diseases have higher value than those targeting less severe diseases	Severe, chronic, progressive, and life-threatening diseases, with multiple medical and nonmedical (i.e. psychological, social) consequences [1–3, 57, 72]Significant impact on caregivers’ QoL [1–3, 69]Lack of knowledge/data on natural history [4]	Addition of subcriteria defining all key aspects of disease severity:Effect on life expectancyEffect on morbidity (includes disability and function)Effect on patient QoLEffect on caregiver QoL
Size of population affected	Alleviate suffering in as many individuals as possible (utility theory)	Interventions benefiting large numbers of individuals have greater value than those benefiting few individuals	Definition of rare diseaseRarity in itself not a prioritization criterion [51, 52, 63, 66]Challenge in obtaining high-quality epidemiological data [5–8]	Specification of scoring scale options with standard definitions of rare and ultra-rare diseases
Unmet needs	Alleviate suffering in individuals with limited alternative interventions (theory of justice)	Interventions for which there are no alternatives or where alternative interventions have major limitations have greater value than those for which there are alternatives that have only minor limitations	Limited number of targeted treatments [3]Limited number of clinicians familiar with a rare condition [3]Limited access to appropriate health services [3]	No adaptation required
Objective: providing large improvements in health outcomes
Comparative effectiveness	Alleviate suffering to the greatest extent (beneficence, deontology)	Interventions that provide major improvements in efficacy/effectiveness have greater value than those with lower efficacy/effectiveness than comparators	Need to define most relevant efficacy/effectiveness outcomes for each disease [9, 59, 61, 73]:Outcomes that reflect natural course of diseaseOutcomes that detect clinically meaningful improvementValue of short-term surrogate outcomes vs. long-term outcomesWell-powered RCTs may sometimes not be feasible for rare diseases [9, 10]. Alternative trial designs, modeling, observational studies and registries may provide clinical data [9, 10, 49, 58]	Disease-specific outcomes modeled as subcriteria
Comparative safety/tolerability	Hippocratic Oath “do no harm” (nonmaleficence, deontology)	Interventions that provide major improvements in safety/tolerability have greater value than those with lower safety/tolerability than comparators	Reduced statistical power of results due to small sample size and small number of trials for a given intervention [59]Risk of rare serious AEs difficult to estimate in a small patient populationLong-term safety issues may not be captured due to short follow-up duration [59]:Need for postmarketing safety data/outcomes registriesNeed to define most relevant safety/tolerability outcomes for each intervention	Disease-specific outcomes modeled as subcriteria
Comparative patient perceived health/PROs	Alleviate suffering as perceived by the patient (Hippocratic Oath « for the good of my patient ») (deontology)Respect for autonomy	Interventions that provide major improvements in PROs have greater value than those with worse PROs than comparators	Limited availability of disease-specific PRO instruments [69]:Not systematically captured in trials; small sample size; short trial duration; absence of relevant comparatorSelection of HRQoL instruments may not be appropriate [69]:Do generic instruments (vs. disease-specific) capture the relevant QoL impact of each disease?Is HRQoL measure validated? Can it detect minimally important differences [69]?	Disease-specific outcomes modeled as subcriteria
Objective: delivering important types of health benefit
Type of preventive benefit	Protect health and prevent suffering (deontology)	Interventions that provide major reductions in disease risks have greater value than those providing no reduction in disease risks	Most rare diseases are genetic [2]; thus, interventions are generally not preventive	No adaptation required
Type of therapeutic benefit	Aim to eliminate rather than merely alleviate suffering (deontology)	Interventions that provide major therapeutic services to the patient (e.g. cure) have greater value than those providing minor services (e.g. relief from a minor disease symptom)	Limited data on type of therapeutic benefit due to lack of long-term studies [59]	No adaptation required
Objective: producing favorable economic consequences
Comparative cost consequences—cost of intervention	Use scarce resources wisely to maximize health from a specific budget perspective (practical wisdom, utilitarianism)	Interventions that reduce treatment costs have greater value than those that increase treatment costs	High cost per patient but budget impact of single product relatively low [18, 50, 67, 71]	Introduction of negative scoring scale
Comparative cost consequences—other medical costs	Use scarce resources wisely from a broad perspective (practical wisdom, utilitarianism)	Interventions that free-up other medical resources have greater value than those that require the use of additional medical resources	Patients with rare disease can use a broad range of medical resources and services, including specialist care, medical examinations, and hospitalization [3]	Addition of subcriteria defining the payer:Other medical costs to healthcare systemMedical cost to patientIntroduction of negative scoring scale
Comparative cost consequences—nonmedical costs	Preserve societal and individual resources wisely from a broad perspective (practical wisdom, utilitarianism)	Interventions that preserve and free-up nonmedical resources have greater value than those that require the use of additional nonmedical resources	Particular need in rare disease to take a broad (i.e. societal) perspective on economic consequences:Economic impact on social services [3]Economic impact of disease and its treatment on patients and their families [2, 70]	Addition of subcriteria defining type of cost and payer:Patient/caregiver productivityCost to wider social care systemNonmedical costs to patientIntroduction of negative scoring scale
Objective: reducing uncertainty through solid knowledge
Quality of evidence	Consider strength of claims about the intervention based on formal evidence (imperative of evidence-based decision making, practical wisdom)	Interventions for which evidence reporting is complete and consistent, relevant to the decision to be made, and valid with respect to international scientific standards have greater value than those for which evidence reporting is incomplete and inconsistent and the evidence has low relevance and validity	Rareness of a condition affects data quality in all areas affecting drug appraisal, including epidemiology [5–8], clinical efficacy and safety [9, 53, 59, 73], and economic consequences	Addition of subcriteria by field of research:Clinical evidenceEpidemiology evidenceEconomic evidence
Expert consensus/clinical practice guidelines	Consider strength of claims about the intervention based on expert knowledge and consensus (practical wisdom)	Interventions strongly recommended on the basis of current expert consensus have greater value than those not recommended by clinical experts	Limited number of clinical experts [1, 64]Availability and quality of clinical guidelines [55]	No adaptation required
Normative contextual objectives: qualitative appraisal
Objective: aligning with the mandate and scope of healthcare system
Mandate and scope of healthcare	Promote and protect health of the population served (utilitarianism, beneficence)	Interventions falling within the scope and mandate of the healthcare system have greater value than those not aligned with these	Improving health of patients with severe diseases is aligned with the mandate of healthcare [68]	No adaptation required
Objective: addressing priorities to increase fairness/justice
Population priority and access	Principle of fairness (theory of justice)	Interventions targeting established priority populations/disease areas have greater value than those not aligned with these established priorities	Rare diseases represent priorities in several healthcare systems at the regulatory policy and societal levels [12, 56, 80, 82, 95, 96]	Development of the transformation process of this objective into two quantitative criteria—required conceptualization of two criteria and design of appropriate scales. This design can be applied when clearly established priorities (e.g. rare diseases) are available in a given context
Objective: aligning with the common goal
Common goal and specific interests	Awareness of stakeholder pressures/barriers helps ensure that decisions are fair-minded and driven by the common goal and not unduly influenced by specific interests (practical wisdom)	Interventions aligned with the common goal have greater value than those aligned with special interests	Highly engaged rare disease patient organizations [47, 48, 54, 62, 65]	No adaptation required
Objective: ensuring environmental sustainability
Environmental impact	Principle of utility and beneficence	Interventions that are produced, used, or implemented without causing environmental damage have greater value than those causing environmental damage	Environmental impact of rare disease treatments expected to be low	No adaptation required
Feasibility contextual objectives: qualitative criteria
Objective: being affordable with low opportunity costs
Opportunity costs and affordability	Principle of efficiency (practical wisdom, utilitarianism)	Interventions are feasible if they are affordable and associated with a low opportunity cost	Need to consider opportunity cost of benefits forgone in other disease areas [56]	No adaptation required
Objective: ensuring and preserving system capacity and appropriate use
System capacity and appropriate use of intervention	Ensure appropriate use of intervention to realize its potential benefit and avoid unintended consequences (practical wisdom, consequentialism)	Interventions are feasible if they can be used appropriately and preserve the healthcare system’s capacity	Lack of local expertise with rare diseases [3]Difficulty of reaching entire target population [3]Monitoring and surveillance requirements [58, 60]	No adaptation required
Objective: having favorable political, historical and cultural context
Political/historical/cultural context	Awareness of political/historical/cultural aspects ensures that decisions are based on a broad understanding of the context (practical wisdom)	Interventions are feasible if their implementation is supported by the political, historical, and cultural context	Development of therapies for rare diseases may broadly advance innovation beyond a single disease area [56]Precedence of decisions	No adaptation required

MCDA multicriteria decision analysis, QoL quality of life, RCTs randomized controlled trials, AEs adverse events, PROs patient-reported outcomes, HRQoL health-related quality of life

Normative Universal Objectives and Quantitative Criteria

These criteria are assessed quantitatively because they are universally quantifiable in the sense that their achievement can be assessed across healthcare systems on scales for which the high and low ends are a priori agreed upon [37].

The objective of addressing areas of high therapeutic need covers three criteria:

Disease Severity: Grounded in the ethical imperative to alleviate suffering in those who are worst off (theory of justice), interventions for more severe diseases have greater value than those for less severe diseases [36]. Rare diseases can have multiple impacts on patients, causing morbidity, disability, reduced quality of life (QoL), and shorter life expectancy [1, 2, 69]. Approximately half of these conditions begin in childhood and many cause disabilities [1], thus posing a high burden on caregivers, usually family members [69], which may have a detrimental impact on their QoL [1-3]. To allow a more differentiated assessment of the multiple domains of disease severity potentially impacted by various conditions, four subcriteria were introduced, including Effect of Disease on Caregivers’ QoL, which is consistent with other assessment frameworks, such as the NICE Interim Process for HST [16].

Size of Population: Based on utility theory, this criterion aims at alleviating suffering in as many individuals as possible [36, 37]. Accordingly, a large population of patients affected represents a high therapeutic need, which is in agreement with surveys of the general public revealing that rareness in itself, apart from other considerations such as disease severity, is not considered a healthcare priority [51, 52, 63, 66]. Capturing an important aspect of social value, this criterion needs to be retained in an appraisal framework that is to be applied across different disease areas. Review of epidemiological data on rare diseases [76] suggested an adaptation of the scoring scale to better reflect the wide range of possible disease incidence/prevalence rates, extending from very common (prevalence >5 in 100) to very rare diseases (<5 in 100,000).

Unmet Needs, whereby a condition for which current interventions have many and serious limitations regarding their effectiveness, safety, tolerability, and QoL impact represents high need [37]. This criterion is highly relevant for rare diseases, where major therapeutic limitations persist and few interventions targeted for a specific condition are available [3].

The objective of providing large improvements in health outcomes is rooted in deontology (i.e. duty- or rule-based ethics) and outlined in the Hippocratic Oath as well in the principles of beneficence, nonmaleficence, and respect of autonomy (principlism) [21, 36, 41]. This is reflected in three criteria:

Comparative effectiveness;

Comparative safety/tolerability;

Comparative patient-reported outcomes (PROs)

One of the issues in rare diseases pertaining to these criteria is uncertainty or lack of consensus on the outcomes that need to be assessed in clinical studies. This issue can be partially addressed by dividing these criteria into subcriteria to represent outcomes specific to the disease, which will allow appraisers to discuss and differentially weight the importance (relevance) of each outcome.

The objective of delivering important types of health benefit is rooted in the ethical imperative to prevent and eliminate suffering (deontology), and covers two criteria [36]:

Type of preventive benefit, whereby preventing a disease entirely (eradication) is the most important type of preventive benefit (risk reduction).

Type of therapeutic benefit, whereby alleviating suffering entirely (cure) is the most important type of therapeutic benefit.

Although the vast majority of rare diseases are of genetic origin [2], and interventions are therefore generally therapeutic rather than preventive, this design acknowledges the value of both prophylactic and therapeutic interventions to ensure compatibility of the framework across therapeutic areas.

Both maximization of health benefits and minimization of costs are legitimate objectives of healthcare decision making, rooted in the ethical imperative of making wise use of scarce resources (practical wisdom) to maximize health under resource constraints (utilitarianism) [36, 37]. Thus, an intervention’s economic consequences represent value attributes and are therefore incorporated into a framework intended to measure value from a holistic perspective, as recommended by the UK Government manual conducting multicriteria decision making [46]. The objective of producing favorable economic consequences is addressed with three criteria; for each of these, cost reduction (or release of resources) is seen as a favorable economic consequence:

Comparative cost consequences—cost of intervention: Net cost of an intervention to the healthcare system in terms of its acquisition, implementation, maintenance, and replacement of existing treatments or interventions.

Comparative cost consequences—other medical costs: All medical cost consequences of an intervention, apart from the cost of the intervention itself (e.g. hospitalization, specialized care, primary care, long-term care, adverse event costs).

Comparative cost consequences—nonmedical costs: All cost consequences outside the scope of medical care.

Interventions can affect patients’ medical or nonmedical expenses through co-payments, travel, and paid caregivers, as well as impact patients’ and caregivers’ ability to work, all of which are highly relevant to rare diseases [2, 3, 70], as well as to many other conditions. Therefore, the adapted framework includes subcriteria, defined by payer (i.e. healthcare system, patients), allowing differentiation of diverse economic consequences and adjustments for appraisals from different perspectives.

Cost effectiveness combines several criteria already covered and is therefore not included in the framework, in line with basic design principles of MCDA to avoid double-counting and with other rare disease frameworks [16, 17, 19].

The objective of reducing uncertainty by grounding decisions in solid knowledge is covered under two criteria, both related to practical wisdom [36]:

Quality of Evidence: This criterion ensures that higher value is placed on interventions that have a statistically solid and unbiased scientific evidence base [36, 37, 77]. Responding to the imperative of evidence-based decision making, this criterion explicitly rewards high-quality research and development. The rareness of a condition affects data quality in multiple areas, including the burden and natural history of the disease, and the health and economic consequences of a proposed therapy [4-8]. For example, designing well-powered, double-blind, randomized controlled trials (RCTs) is challenging for rare diseases because patients are few [9] and effective comparators are often absent [10]. Therefore, real-life studies and modeling can play an important role for assessing rare disease therapies [17]. Thus, the adapted framework includes subcriteria representing different fields of research, allowing a differential assessment of data quality.

Expert Consensus/Clinical Practice Guidelines (CPGs): This criterion captures the strength of recommendations for optimal clinical practice based on expert interpretation of current scientific knowledge as well as bedside expertise [37]. A strong recommendation in a CPG developed following AGREE principles [78] raises confidence in the value of an intervention. For rare diseases, the availability and quality of CPGs is often limited [55], thus it is particularly important to capture expert knowledge, bearing in mind the scarcity of clinical experts and clinical data [1, 64].

Transformation of a Generic Qualitative Criterion into a Quantitative Context-Specific Criterion: Population Priority and Access

The objective of Addressing Priorities to Increase Fairness/Justice reflects a society’s sense of justice, which may entail giving priority to certain populations, as defined in specific societies and healthcare systems [36, 37]. Although there are various models of distributive justice (e.g. libertarian, communitarian, egalitarian, and utilitarian) [79], solidarity with those most in need is a key concept [56], as expressed in surveys of the general European population [51, 63]. Patients with rare diseases may be seen as a particularly disadvantaged population as they are often affected by social isolation [1, 57], delayed diagnosis [1, 3, 72], and inequalities in access to adequate treatment and care [1]. The disadvantaged status of these patients is acknowledged in regulatory policies that support research in orphan diseases [11, 12]. In addition, a number of public initiatives were launched to improve delivery of healthcare and social services to patients with rare diseases [80-83].

Although most decision makers consider population priorities [28], specific priorities may or may not be explicitly established in the context of a given healthcare system. Thus, one can distinguish two types of contexts:

If no population priorities are explicitly established, this objective cannot be quantitatively operationalized and must remain in the qualitative part for the framework for qualitative, general (case-by-case) consideration.

If specific population priorities are clearly established, the framework is adapted to integrate the objective of addressing priorities into the quantitative part of the framework, bearing in mind that multiple priorities may exist in a given context. To operationalize this integration, each population priority is defined as a criterion:

priority 1 (e.g. rare diseases)

priority 2 (e.g. HIV)

priority 3 (e.g. diabetes)

etc.

This design allows relative weighting of each priority versus other, potentially competing priorities that may have been established. Scoring is performed based on how well the intervention to be appraised is aligned with the respective priority. The output of this design can thus quantitatively capture the contribution of priorities (in contexts where such have been established) to the value of an intervention. In addition, this design allows addressing variability in country or institution policies with respect to prioritization of specific disease areas.

Normative Contextual Objectives and Qualitative Criteria

These criteria cannot be operationalized on a universal basis and are therefore appraised qualitatively in the generic framework [37]. As illustrated in the section above, when adapting the framework to a given context, objectives can be more closely defined and made operationalizable into quantitative criteria, thereby allowing their integration into the quantitative appraisal.

The objective of Aligning with the Mandate and Scope of the Healthcare System rests upon the principle of beneficence and utility, and its inclusion in the framework ensures that this is explicitly considered in each appraisal [36, 37]. The principal mandate of healthcare is to restore and maintain normal functioning [68], and interventions for rare diseases with serious health effects are generally aligned with this mandate.

This objective of Aligning with the Common Goal (i.e. the health of patients, populations, and healthcare systems) rather than with special interests corresponds to the reasonableness condition of the Accountability for Reasonableness framework [23, 24], which demands that decisions be based on relevant and mutually acceptable reasons [36]. Awareness of stakeholder pressures and barriers helps ensure that decisions are fair-minded and not unduly influenced by special interests [37]. For rare diseases, several organizations exist that inform, support, and advocate for patients [47, 48, 54, 62, 65]. While these are important to bring real-life patient perspectives to decision making, a balanced approach needs to be exercised for fair allocation of resources across populations in need.

Although reducing Environmental Impact is a universal normative objective, it currently does not play a significant role in healthcare decision making (although many decision makers think that it should [28]), and is thus appraised qualitatively, but could be incorporated into the quantitative (universal) part of the framework in a given context [37]. Due to their rarity, the environmental impact of rare disease treatments is expected to be low.

Contextual Feasibility Objectives and Qualitative Criteria

Some interventions may be desirable from a normative point of view, but not feasible in a given context.

Consideration of Affordability and Opportunity Costs, i.e. resources or existing interventions that may be forgone if a new intervention is adopted, is aligned with the principle of efficiency and utilitarianism (maximize health resources) [43], which comes into play at both the patient and societal levels [36, 37]. While the normative aspect of economic considerations (i.e. favorable economic consequences) is included in the quantitative (universal) part of the framework, affordability and opportunity cost considerations require a financial/budgeting exercise to determine feasibility in a given context [37]. For rare diseases, this reflects on whether resources foregone in other disease areas (e.g. more common diseases) are significant in the context of system-wide healthcare economics [56].

The ability of a healthcare system to ensure appropriate use of a new intervention and realize its potential benefit within the system’s capacity, depends on infrastructure, organization, skills, legislation, barriers, and risk of inappropriate use [37]. Relevant for rare diseases is to ensure that specific skills, infrastructure, and surveillance requirements (e.g. patient registries, monitoring) and the ability to reach remote populations are systematically considered [3, 58, 60].

The political, historical and cultural context is important to consider in appraisals. This includes the legal framework, such as, for example, the Equality Act, under which the NICE is required to avoid discrimination based on protected characteristics, and the Human Rights Act, under which it must consider implications for human rights [84]. This objective also covers the impact of the intervention on innovation and research, an important aspect for rare diseases, as research in this area was shown to advance understanding of pathophysiology, broadly contributing to the development of knowledge [56]. Precedence is another important consideration in coverage decision making in the rare disease field, as in other therapeutic areas.

Hypothetical Example for Application of the Adapted Framework

Table 2 shows an example of the application of the adapted framework in two appraisal contexts: if population priorities have not been established (context A), they are not part of the quantitative model and are considered qualitatively. In contrast, in a context in which population priorities have been explicitly defined (context B), these are incorporated into the quantitative model. In this case, the appraising committee must decide how much relative weight they will trade off from the other domains to ‘population priorities’ and how they will weigh the priority ‘rare diseases’ against other priorities established in their context (Table 2a).

Table 2

Example of the application of the adapted EVIDEM MCDA framework to appraising a hypothetical intervention in two contexts: ‘population priorities’ not explicitly established and thus not part of the quantitative model, and ‘population priorities’ established and thus part of the quantitative model

Domains	Domain weights		Criteria			Criteria weights	Normalized criteria weights
	No explicit priorities	Priorities established					No explicit priorities	Priorities established
(a) Relative hierarchical weighting of the domains and criteria of the quantitative MCDA model (independent of the hypothetical intervention), representing the value system and trade-offs of the committee
Normative universal criteria
Need for intervention	30	30	Disease severity			30	0.09	0.09
			Size of affected population			20	0.06	0.06
			Unmet needs			50	0.15	0.15
Comparative outcomes of intervention	30	28	Comparative effectiveness			50	0.15	0.14
			Comparative safety/tolerability			30	0.09	0.08
			Comparative patient-perceived health/PROs			20	0.06	0.06
Type of benefit of intervention	10	10	Type of preventive benefit			30	0.03	0.03
			Type of therapeutic benefit			70	0.07	0.07
Economic consequences of intervention	20	17	Comparative cost consequences—cost of intervention			50	0.10	0.09
			Comparative cost consequences—other medical costs			25	0.05	0.04
			Comparative cost consequences—nonmedical costs			25	0.05	0.04
Knowledge on intervention	10	5	Quality of evidence			50	0.05	0.03
			Expert consensus/CPGs			50	0.05	0.03
Contextual criteria			Country policy
Population priorities	–	10a	Rare diseases	Second priority of national health		40a	Assessed qualitatively	0.04
			Other priorities	Cancer and mental disorders first and third priority		60a		0.06
Sum	100	100				–	1.00	1.00

For simplicity, subcriteria are omitted in this example

MCDA multicriteria decision analysis, AE adverse event, CPGs clinical practice guidelines, PROs patient-reported outcomes, QoL quality of life, RCT randomized controlled trial

aThe weights reflect the established priorities in the country

bScale: −5 to 5 for comparative criteria; all other criteria (noncomparative) are scored on a scale of 0–5

cValue contribution = normalized weight (W x) × standardized score (S x, assigned score divided by maximum 5)

dAssessed qualitatively in this context

eValue estimate = ∑(W x × S x). The maximum (1) corresponds to a hypothetical intervention that prevents and cures severe endemic diseases with significant unmet needs and which, compared with existing approaches, has demonstrated large improvements in efficacy, safety, and PROs, as well as positive economic consequences. (When specific healthcare system priorities are included in the quantitative model, this definition also includes full alignment of the intervention with these priorities.)

Now let us consider assessment of a hypothetical intervention X for a rare condition Y by the same committee (Table 2b). Committee members assign a performance score for each criterion that expresses their judgment of the available evidence, as shown in the example. Scores are standardized and multiplied by the normalized weights (in Table 2a) to calculate the contribution of each criterion to the value of intervention X (Table 2b). The overall value estimate can then be used to rank interventions across disease areas. In this hypothetical example, inclusion of ‘population priorities’ in the quantitative part of the model increases the value estimate of X due to its full alignment with the rare disease priority and the relatively high weight the committee has assigned to this criterion. The committee also qualitatively considers what impact contextual criteria may have on the value of intervention X, which may impact its ranking of intervention X. If priorities have not been included in the quantitative value estimate, their potential impact can be captured qualitatively and may affect ranking.

Discussion

In this study, we analyzed the EVIDEM framework from the perspective of issues raised by rare diseases and developed it further in response to these issues. The adapted EVIDEM framework provides an operationalizable platform to integrate individual and social values, competing ethical dilemmas, and uncertainty, which are particularly challenging in appraising interventions for rare diseases. The addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes and economic consequences of interventions, inclusion of negative scoring scales for all comparative criteria, and integration of a methodology to further incorporate context-specific population priorities and policies makes the EVIDEM framework particularly responsive to rare diseases, while maintaining its applicability across disease areas.

Development of the EVIDEM framework involves continuous research and development on handling of uncertainty, which is particularly pertinent to the context of rare diseases where ethical dilemmas and variations in judgments and perspectives are challenging for the healthcare community [25-27]. Uncertainty in judgments on evidence can be explored using score ranges [85]; uncertainty in weights can be gauged through application of different weighting techniques (e.g. pair-wise comparison) [30, 46, 75]; uncertainty due to variability in individual perspectives (weights) and judgments (scores) can be assessed through standard measures of statistical dispersion [25–27, 32, 33]; uncertainty due to model structure can be explored by modifying the structure, e.g. eliminating the lowest weighted criteria [86]; and reproducibility can be examined by repeating the appraisal exercise (test–retest) [33, 87]. In this study, we additionally developed a method to address variability in country or institution policies with respect to prioritization of specific disease areas. This method allows adapting the framework according to whether or not explicit prioritization policies exist and to examine the relative importance of specific priorities.

In a survey of the Norwegian population, the notion that “Patients with rare diseases should have the same right to treatment as others even if more expensive” found strong support [51]. Similarly, in a UK survey, concern for fairness exceeded concern for population health maximization (i.e. utility theory), the guiding principle of the cost-effectiveness-based approach to prioritization [63]. Cost effectiveness is not considered in current [16, 17] or proposed rare disease frameworks [18–20, 68], and several HTA agencies waive their requirement for cost-effectiveness analysis for orphan drugs [13, 88]. Some propose modifying the current cost-effectiveness paradigm by, for example, assigning quality-adjusted life-year (QALY) weights depending on disease prevalence [9] or different willingness-to-pay thresholds for diseases with high ‘social value’ [89]. However, people do not prioritize rareness but severe, life-threatening diseases, with high unmet needs and interventions that reduce reliance on informal caregivers [63, 66]. These attributes, while characteristic of many rare diseases [1], also apply to other disease areas. Implicit in these proposals is the realization that there is a need for a holistic definition of value, beyond the cost per QALY, to guide appraisal of healthcare interventions.

In comparison to other proposed MCDA frameworks for rare diseases, where selection of decision criteria was based on a literature review, consultation with stakeholders, or rationales [18-20], selection of decision criteria for the EVIDEM MCDA framework is additionally explicitly rooted in a comprehensive set of ethical principles as well as MCDA design principles, which promotes thorough exploration of the values underpinning appraisal. In addition, due to alignment with the ultimate common goal of all healthcare stakeholders, i.e. identifying interventions that are beneficial to the health of patients as well as to populations and healthcare systems (definition of most valuable interventions), the design proposed here can be applied system-wide across interventions and diseases, in distinction to other MCDA rare disease frameworks. System-wide operationalization is further supported through fully developed implementation methods, particularly the generic design of the scoring scales, which measure evaluators’ judgment of the available evidence across types of interventions and outcomes.

This approach is also in contrast to MCDA models that are designed adhoc to address a specific decision problem, with a discrete set of known options (i.e. interventions). Such ad hoc models can be designed around these options and their attributes, in terms of criteria selection and scoring scales [46, 90–92]. Ad hoc models, while useful for a circumscribed decision problem, are not adapted to the resource allocation context in which decision makers need to decide, in a consistent manner, whether any proposed intervention provides sufficient value to be funded and adopted into practice. In such contexts, decision makers need a tool to measure value grounded in fundamental principles that reflect the goals of the healthcare system [93]. For example, such an MCDA tool, including decision criteria such as incremental benefits, incremental total cost, and quality of evidence, was proposed for Israel’s Public National Advisory Committee [77]. In addition, EVIDEM has been adapted and implemented in several jurisdictions as a system-wide approach that encompasses all the aspects of decisions for resource allocation [34, 35].

Decision making at its core involves the balancing of ethical dilemmas. Aiming to maximize the overall utility of society (utilitarianism), few resources should be allocated to therapies benefiting few individuals [56]. From a rights-based perspective, access to appropriate healthcare is a right, which is constitutionally established in some jurisdictions, although its scope is open to interpretation [56]. Obviously, a rights-based approach needs to be counterbalanced with considerations of healthcare efficiency and sustainability. The multicriteria framework proposed here makes this balancing act explicit by defining decision criteria, each providing a distinct contribution towards the goal of healthcare. Prioritization of rare diseases is made explicit and weighted against other priorities and the goal of benefiting as many people as possible, while taking into account disease severity, unmet medical needs and economic consequences. This MCDA design incorporates the moral principles of beneficence, nonmaleficence, respect for autonomy, and distributive justice, set forth by Beauchamp and Childress [39, 40] and explicitly subscribed to by the NICE [94], and ensures that these principles are actively considered in decision making. Such approaches facilitate communication to achieve workable resolution of ethical dilemmas across stakeholders, which is necessary to promote the most valuable healthcare interventions to optimize health of patients, populations, and healthcare systems.

Conclusion

The adapted framework measures value in its widest sense and is responsive to rare diseases issues and policies. It provides an operationalizable platform to explore values, competing ethical dilemmas, and uncertainty in appraising healthcare interventions.

Supplementary material 1 (PDF 173 kb)

The adapted EVIDEM framework provides an operationalizable platform to integrate individual and social values, competing ethical dilemmas, and uncertainty, which are particularly challenging in appraising interventions for rare diseases.

The addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes and economic consequences of interventions, inclusion of negative scoring scales for all comparative criteria, and integration of a methodology to further incorporate context-specific population priorities and policies makes the EVIDEM framework particularly responsive to rare diseases, while maintaining its applicability across disease areas.

Additionally, a method was developed to address variability in country or institution policies with respect to prioritization of specific disease areas.