A Critical Examination for the Pricing of Eculizumab and Efgartigimod in Myasthenia Gravis.

The purpose of this commentary is to focus on the downside of assumption-driven simulation modeling, the potential creation of a multitude of competing models, the mathematically impossible quality adjusted life year (QALY) and the failure to observe the axioms of fundamental measurement in mapping ordinal EQ-5D-5L preferences from the ordinal Quantitative Myasthenia Gravis (QMG) score. A second aspect of this commentary is to propose standards that should be set for the creation and evaluation of value claims in health technology assessment, in particular need fulfillment quality of life (QoL), that meet the demarcation test to distinguish science from non-science. The result is that the present ICER pricing claims for eculizumab and efgartigimod in myasthenia gravis should not be applied without consideration of more relevant evidence. © Individual authors.

INTRODUCTION

The manifest deficiencies in health technology assessment has been extensively documented [1 2]. It has been described as a meme, with high transmission fidelity; one that has persisted for some 30 years because of a decision to invent evidence (described as ‘approximate information’) to support claims for cost effectiveness at product launch [3]. Given the option of establishing provisional pricing and access criteria, subject to a targeted real world evidence research program to fill in gaps and provide an ongoing framework for disease area and therapeutic class reviews, health technology assessment has instead favored modeling and the creation of evidence to support cost-effectiveness claims at product launch.[4]

The recently released ICER final evidence report for eculizumab (Soliris; Alexion Pharma, a subsidiary of AstraZeneca) and efgartigimod (Argenx) for the treatment of myasthenia gravis is an example of the traditional form of health technology assessment which unfortunately denies the standards of normal science and, in particular, the axioms of fundamental measurement. It is in the relativist tradition of claiming that truth is consensus and that we are in no position to judge the merits of one analytical framework over another [4]. In this world view, all theoretical and conceptual frameworks are equally valid. In its most extreme form, ‘social constructionism’, there are no facts out there to be discovered, only invented. There is no mind-independent reality or any attempt to discover it; there is no incentive to seek facts, which are not there to be discovered.

The purpose of this commentary is: first to outline the failure of the ICER reference case approach to meet the required standards of normal science and, second, to propose a minimum set of evidentiary standards to support value clams by manufacturers. This commentary is in five parts: first, a brief overview of the unreality of the ICER simulated model; second, a critique of the ICER simulated worlds with particular reference to the creation of imaginary preference scores; third, a brief overview of the required scientific standards for value claim creation; fourth the potential multiverse of assumption driven imaginary simulations and claims; and, finally, the contribution of Rasch Measurement Theory to establish a new paradigm for measurement and evaluation of value claims, in particular need fulfillment quality of life (N-QoL).

THE UNREALITY OF ICER BELIEF

ICER appears to ascribe to a relativistic world, where evidence is created and not discovered. This embrace of relativism shortchanges both patients and manufacturers. We must make the effort, not only to emphasize that science is concerned with the discovery of new yet provisional facts through a modified process of conjecture and refutation, but that we seek to evaluate credible, empirically evaluable and replicable competing therapeutic claims. At the same time we must emphasize that assumption driven lifetime simulations are an analytical dead end. We have to dispel an elementary logical error to justify choice of assumption in the ICER simulation model. This violates a simple point that was made by the Scottish philosopher David Hume (1711-76), the problem of induction [5]. As Magee puts it: an assumption cannot be established by observation, since we cannot observe future events. And it cannot be established by logical argument, since from the fact that all past futures have resembled past pasts it does not follow that all future futures will resemble future pasts[6]. ICER cannot claim any superiority for its modelled claims through its choice of ‘realistic’ assumptions.

Central to this process of discovery is measurement; unless our instruments for evaluating response to therapy meet the required standards we can make no claim for the superiority of one therapy over another. This requires attention to the importance of distinguishing ordinal from ratio scales; or even the distinction between ratio and interval scales. If we don’t, we end up believing that the EQ-5D-5L preference scores are actually ratio scales in disguise.

It is into this imaginary world of assumption-driven simulations that manufacturers are cast once ICER decides to make an example of their product. In all fairness, manufacturers are all too often on the back foot and there is nowhere to turn to challenge ICER from day one [7]. An example is seen in the Health Benefit Price Benchmark (HBPB). This relies entirely on assumption and false belief in measurement theory. As will be noted here, there can be a multitude of HBPB’s each driven by different models and assumptions in the same disease area and for the same therapies.

THE SIMULATED ICER IMAGINATION

The apparent purpose of the ICER modelling is to invent the case for the imaginary cost-effectiveness of eculizumab and efgartogimod with each added to conventional therapy versus conventional therapy alone. The base case analysis to invent evidence for cost-effectiveness claims used a two-year, four state Markov model where a simulated cohort of hypothetical patients enters the simulated imaginary model, assumed to be unimproved on initial treatment, receiving the a new product plus conventional therapy or conventional therapy alone.

Central to simulation modeling is the belief in the quality adjusted life year (QALY) which, unfortunately, has been demonstrated to be a mathematical impossibility. The QALY is constructed by multiplying simulated time spent in a health state by a preference score, usually multiattribute, on a scale from 0 = death to 1 = perfect health. In practice all multiattribute scores produce negative values which invalidates their use in QALY creation. To create a QALY you require preference scores with ratio properties. That is they have a true zero and a scale with invariance of comparisons.

In this case preference scores for the imaginary simulation were taken from a mapping of the QMG to the EQ-5D-5L. The procedure was

Health state utilities were derived from a deidentified data source … In the dataset, the QMG and EuroQoL EQ-5D-5L states were reported for a cohort of 257 patients with gMG. Utility was determined using the EQ-5D-5L health states and the US-based societal value set … The association between QMG and EQ5D-5L was estimated using a univariate linear regression model, including 252 patients with complete QMG scores. The model estimated that patients with a QMG score of “0” had a starting utility of 0.97 and that each 1-point increase in QMG score was associated with a 0.03 decrease in utility.

The Quantitative Myasthenia Gravis (QMG) score is a 13-item scale used to quantify disease severity in myasthenia gravis [8]. The scale measures ocular, bulbar, respiratory, and limb function, grading each finding, and ranges from 0 (no myasthenic findings) to 39 (maximal myasthenic deficits) by collapsing these various scores to a single value. It is an ordinal multivariate score which ranks respondents by the presence and severity of the disease. The QMG lacks invariance of comparisons between the scores; it cannot support parametric statistical analysis involving any standard arithmetic operation: addition, subtraction, multiplication or division. Only non-parametric statistics are valid, but even then the fact that the scores are multiattribute in attempting to capture a range of clinical markers means that the QMG lacks dimensional homogeneity or unidimensionality and construct validity. As such, it cannot serve as the basis for mapping to a preference score.

The univariate regression model employed in the attempt to create preference scores is invalid. The scores produced are meaningless. A claim that each 1-point increase in QMG score led to a 0.03 decrease in utility from a starting utility of 0.97 is untenable. The authors also fail to realize that their target preference score the EQ-5D-5L is also an ordinal score. If they insist on claiming that they, in some alternate reality, can successfully map to the EQ-5D-5L then they should also recognize that, for US valuations, 20% of health states yield a negative score (‘states worse than death’) [9].

It is worth noting that another myasthenia gravis instrument, the MG-QOL15r, which purports to measure quality of life (or at least health related quality of life) also fails to meet the standards required to evaluate response to therapy [10]. Again, this is an ordinal scale.

The QALYs presented for the modeling are mathematically impossible and the entire simulation exercise being not applicable. This is the inevitable outcome of preference scores that have ordinal properties; a measurement characteristic of all direct and indirect preference scores and the majority of patient reported outcome (PRO) instruments. There is no apparent application of the standards of normal science and the role of the axioms of fundamental measurement [11].

REQUIRED SCIENTIFIC STANDARDS

Formulary decisions must rest upon evaluable claims for therapy impact, notably comparative claims that are consistent with the standards of normal science and the axioms of fundamental measurement. Instead, after 30 plus years of health technology assessment we face exactly the opposite commitment: therapy claims that are contrived in their focus on inventing evidence and the implicit rejection of any concern with meeting the standards of fundamental measurement. The multiattribute preference score entered center stage with the QALY acclaimed as the only valid construct to support modelling with pricing and access recommendations. Even at this stage concerns were expressed that there were multiple preference scores each creating their own QALY which would lead to competing cost-effectiveness claims. This issue remains unresolved. We now realize with the benefit of hindsight and a better appreciation of the measurement standards that apply in the physical sciences and the mainstream social sciences such as education and economics, that the decision in favor of approximate information in lieu of hypothesis testing to generate new evidence, was a setback for a commitment to the standards of normal science.

The assumption-driven simulation model produces seven imaginary outcomes; value claims that are not credible, empirically evaluable or replicable. The main ersatz value claims or outcomes are:

Total drug costs

Total costs

Quality Adjusted Life Years (QALYs)

Life years

Expected value of life years (evLY)

Expected value of life years gained (evLYG)

Health Benefit Price Benchmark (HBPB)

If a simulation model is designed to project claims for years and even decades into the future, then it falls at the first hurdle to meet the standards of normal science in failing to produce empirically evaluable claims.

The drug and total costs are a patchwork of assumptions built on present and prior costs and the claimed ‘cost-effectiveness’ is not anchored to any unit cost classification (e.g., CPT codes). Life years are the product of Markov states and the transition probabilities association with those states (including the one- way transition to death). Any redesign of the Markov states and transition probabilities will lead to changes in life year claims. The life years are entirely imaginary, created by the assumption driven simulation model.

A commitment to the mathematically impossible QALY dooms both aggregate estimates of lifetime QALYs and cost-per-QALY thresholds but also the equal value of life year metrics (evLY and evLYG) as QALY estimates are integral to both measures. If one assumption (or belief) dooms the QALY based claims, it the insistence that ordinal preference scales generated by direct and indirect preference instruments are ratio scales in disguise.

HEALTH BENEFIT PRICE BENCHMARK

Perhaps the prime example of a false claim is the Health Benefit Price Benchmark (HBPB). This is operationalized as the highest price a manufacturer should charge for a treatment.

The eugenic implications for access to and denial of care are clear [12]. This highest price is based on the amount of improvement in overall health (defined by the preference score attributes) patients receive from that treatment, when a higher price would cause disproportionately greater losses in health among other patients in the health system due to rising overall costs of health care and health insurance. In short, it is the top price range at which a health system can reward innovation and better health for patients without doing more harm than good. The fatal flaw is that the entire exercise is based on a failure to recognize the standards of normal science, notably the axioms of fundamental measurement, and a belief that the imaginary QALY can support health care allocation decisions. However, you cannot ‘adjust’ preference scores by each other if you are trying to standardize for age and gender differences between health states. It is disallowed as the preference score is ordinal. Health care resource allocation cannot be based on imaginary constructs. The HBPB is meaningless, implying as it does that some health states, from a community preference for health attributes perspective, are more ‘worthy’ of support than others.

THE IMAGINARY MULTIVERSE

If modeled simulated assumption driven claims can be invented under one scenario, then they can be contrasted to a potential multitude of other modeled scenario claims given a change in assumptions. There is only one caveat: unless one can claim that their assumption choice sets them aside from any other potential model, either now or in the future, they are just one among many. Assumptions for the future cannot be justified by reference to the past. Irrespective of the claim that can be challenged and the imaginary claims disputed (but not in terms of real word outcomes), we are reduced to a debate over assumptions. Sensitivity analyses, let alone that technical favorite, probabilistic sensitivity analysis, will not save the day; any one of the potential for thousands of other modeled claims can be defended in precisely the same terms.

Interestingly, there is the option of multiple models with the release of the cloud ICERAnalytics software system [13]. Ostensibly defended in terms of transparency and the ability of decision makers in a health care system to assess (tweak) the impact of changing model parameters (i.e. assumptions) in the model that supported claims for denial of care and restricted access criteria. The possibility, which will no doubt be exploited, is to reverse ICER claims by a judicious choice of assumptions and to hoist ICER with its own petard.

If there are an infinite, or at least, a potential multiverse of modeled imaginary claims then we will never be able to create and compare therapy options. In the case of eculizumab and efgartogimod any ICER claim cannot be considered even provisional; just one of a multiverse of competing model claims which can each produce an infinite series of progeny, each of which can then produce a series of progeny through changing assumptions, each with its impossible HBPB claim. This places the model in a bind: one cannot claim a unique status for their model because there are no criteria for uniqueness that will apply now and for all future modifications of this model.

The result of this assumption-driven Markov simulation is the imaginary claim that, on the basis of a series of assumptions, set the pseudoscientific stage for recommending a discount from the Federal Supply Schedule to achieve a HBPB price range for both products. In the case of eculizumab the ICER imaginary annual price to achieve a $150,000 QALY was $19,400 while for efgartigimod the annual price was $28,400. These are entirely imaginary prices resting on a series of simulated model assumptions and a QALY that is an impossible mathematical construct. Different assumptions could create simulations with entirely different annual prices as long as we are prepared to assume the QALY is not an impossible mathematical construct.

These pricing claims are, unfortunately, taken seriously and by the time a coherent evidence based argument is in place, the damage would be done for patients and caregivers in the denial of care in myasthenia gravis.

RASCH MEASUREMENT AND STATISTICAL MODELLING

In the social sciences statistical modelling is the dominant analytical technique to describe a data set. The object is to fit the model to the data, if necessary by the rejection of potential explanatory variables. This stands in contrast to the physical sciences where the measurement task is to obtain data that fit the model. The requirements of the model, construct or trait that is to be measured drive data collection and item selection. The distinction is between exploratory/descriptive models, fitted to the data (e.g., econometric modelling), and confirmatory/predictive models utilizing probabilistic conjoint measurement, where the requirement is for the data to fit the model. It is this latter approach that drives the Rasch model [14].

Consideration of the Rasch model leads to the required measurement properties of the instrument. In human subject research where the objective is to measure latent traits. We must start with a substantive theory about what it is we are trying to measure. Item development and selection must be driven by our knowledge of the latent trait. It may turn out that the latent trait is not actually quantifiable. The Rasch model tests the hypothesis that we are measuring a quantitative unidimensional latent trait. Two questions are central to this: (i) how well do the empirical data fit the measurement model requirements and (ii) does the instrument yield invariant interval-level measures for the intended purposes?

The attractive features of measurement in Rasch modeling – unidimensionality with linear, additive, invariant values on an interval-level measurement scale – exist only to the extent that the data fit the Rasch model requirements; guided, of course, by an understanding of how the latent trait will be captured in practice. No other patient reported outcome instrument, whether generic or disease specific can meet these requirements for fundamental measurement. They are locked into a paradigm that dismisses the required axioms of fundamental measurement, relying on the belief in the primacy of data over substantive theory; the notion of quality control in the selected data does not arise. We have to use all the data regardless of quality and measurement properties. It is Rasch measurement that underpins the next generation quality of life claims.

NEXT GENERATION QUALITY OF LIFE

Rejecting invented evidence also means rejecting ordinal multiattribute preference scores and the QALY. Both are well past their use by date; indeed, if they ever had one in the first place [15]. Avoiding community preferences for health states defined in terms of a bundle of symptoms and functions, does not mean that the next generation QoL measures ignore clinical symptoms and functional status. The potential contribution of these attributes is seen through the lens of the patient (or caregiver) as elements in a broader holistic framework. As the patient (or caregiver) is the ultimate beneficiary of a therapy intervention the value claim, expressed as QoL, focuses on the need of the patient and the extent to which that need is fulfilled. It is the benefit a patient derives from an intervention specific to a disease state defined in the patient’s own terms that is the single relevant attribute.

Need fulfillment QoL measures based on Rasch Measurement Theory are not new; they have just been ignored in favor of ordinal multiattribute preference measures. Developed over the past 25 years for specific chronic disease states there are now some 30 disease states covered (including: atopic dermatitis, psoriasis, growth hormone deficiency, Crohn’s disease, depression, asthma, COPD, sickle cell disease, herpes, ulcerative colitis). There is no measure for myasthenia gravis. As disease specific measures the need fulfillment measure captures the overall impact of living with a particular disease from the patient’s perspective. This provides the framework for evaluating the extent to which patient (or caregiver) need is met with competing therapy interventions. The items selected for each instrument are subject to an extended process of item selection through the application of Rasch Measurement Theory. Items finally selected are ranked in terms of the difficulty of a need being met and the ability of the respondent to meet that need expressed in probabilistic terms. The number of items selected is relatively small, typically in the range 25-30. The instrument can be completed in 4 or 5 minutes.

This single index of patient value can be transformed to a bounded ratio scale that is unique to each disease state. This creates the Need-QOL (or N-QOL) measure, which is robust and accurate, meeting all the required standards detailed above [16]. As the N-QOL is on a bounded ratio scale in the range 0 = no needs are met to 1 = all needs are met it can be used to create need-based quality of life claims by multiplying time in a disease stage by the N-QOL score to create the N-QAL. By design, negative values are impossible; scores for different instruments across disease states can be compared.

MINIMUM STANDARDS FOR VALUE CLAIMS

There are six standards that must be met for credible and evaluable value claims, including clinical endpoints, patient reported outcomes (PROs), QoL and resource utilization. Failing to meet any one of these standards means the value claim must be rejected. In many cases claims will have already ratio properties based on agreed clinical measurement together with measurable (e.g., CPT code) claims for resource allocation impact; costs are not an acceptable claim. The focus of Rasch measurement, as the only acceptable analytical framework is, of course, focused on latent constructs of which quality of life is probably the most relevant.

MEETING THE STANDARDS OF NORMAL SCIENCE

The single most important standard is to meet the requirements of normal science: All value claims must be credible, evaluable and replicable. If not, like the QALY, the claim is nothing more than pseudoscience and must be rejected. Invented value claims have been the mainstay of health technology assessment for 30 years; to overcome this will be difficult.

SUBMITTING VALUE CLAIM PROTOCOLS

Manufacturers and others submitting value claims must demonstrate how the claim is to be evaluated: All value claims must be accompanied by an evaluation protocol. Failure to provide a claims evaluation protocol must lead to a rejection of the claim.

RECOGNIZING THE AXIOMS OF FUNDAMENTAL MEASUREMENT

All value claims must conform to fundamental measurement standards; this means that the claim submitted must have ratio measurement properties with a true zero; if this is impossible, then the fallback is an interval scale.

SUBMITTING SINGLE ATTRIBUTE CLAIMS

Following the standards of measurement for physical science, all value claims should be for a single attribute whether this is for clinical, outcomes, PROs, quality of life or resource utilization: value claims must be for single attributes defined by a ratio scale meeting requirements for construct validity, content validity and unidimensionality.

SUBMITTING DISEASE SPECIFIC CLAIMS

As the patient (or caregiver) is the presumed beneficiary of therapy intervention, value claims to support that intervention must be specific to a target patient population within a disease area.

REPORTING VALUE CLAIM EVALUATIONS

Value claims must, in the case of formulary submissions, be evaluated and reported to the formulary committee or other health system decision makers in a reasonable or meaningful time frame.

OVERVIEW: A PARADIGM SHIFT?

To accept the imaginary simulations as critical inputs to formulary decision-making and social prices requires a major suspension of the standards of normal science and the axioms of fundamental measurement. Unfortunately, this relativistic belief that no one system of ‘truth’ is superior to another and that no one source of knowledge is superior to another, is an article of faith. To a relativist, we cannot make claim to superior evidence; alternative belief systems as an analytical framework are equally valid. For relativists, science is not necessary to come to grips with reality; any belief system will suffice to make a decision. Evidence for simulation models is never discovered but constructed within a social community in health technology assessment. Its laurels rest on rhetoric, persuasion and authority. This is the antithesis of what science does: to show that a consensus view must be abandoned when it is at odds with accepted scientific standards and the evidence.

There will be pushback; a belief system is not overturned by logic and demonstration. Abandoning a belief, a faith, based upon imaginary constructs is difficult. What Dawkins describes as a mind virus is tenacious in its hold on analysts [17]. As a first defense of the belief system will be the plaintive: everyone does it. However, claims must be credible, evaluable and replicable not judged by some variant of probabilistic sensitivity analysis within a blinkered view of an assumed relevant simulation. Perhaps, as noted in previous commentaries, belief is strongest when the object of that belief is clearly impossible: Certum est quia impossibile est.

There is a need for a commitment to a deeper understanding of therapy impact, of the contribution of new therapies as part of a structured research program to uncover new, yet provisional facts in myasthenia gravis. The denial of hypothesis creation and assessment is a barrier to new hypotheses; accepting imaginary claims to support pricing reductions and denial of care can discourage further activities in disease areas. It is not just that discovery is put to one side but of denying that discovery has any role.

CONCLUSIONS

We must abandon the search for a single value Holy Grail to drive formulary decisions, with acceptance or denial of care based on assumption driven simulations. We must base decisions on attributes specific to a disease state and established by formulary committees. Factoring in a range of attributes with required measurement properties together with input from patients themselves should be sufficient to negotiate an acceptable provisional price and conditions for access to care. These can be modified over time as new data become available as part of ongoing disease area and therapeutic class reviews.

The advent of the disease specific N-QOL means the end of multiattribute ordinal preference scores and the impossible QALY. This provides an assured basis for value claims that represent the need of patients (and caregivers) and a measure of the extent to which that need is met. The key development that has made this possible is the ability, recently developed, to transform a single index of patient value based on Rasch Measurement Theory to a bounded ratio scale with all necessary properties to evaluate need and its determinants, as well as robust and accurate measures of therapy response.

The next step, given the number of need fulfillment instruments already developed, is to initiate a research program to evaluate need in these diverse disease areas, supported by trials and observational studies to create value claims for therapy interventions. There is no longer any need to invent evidence for non-evaluable QALY claims. All we can hope is that patients in myasthenia gravis are not adversely impacted in the meantime.

Conflicts of Interest: PCL is an Advisory Board Member and Consultant to the Institute for Patient Access and Affordability, a program of Patients Rising.

The opinions contained in the paper are those of the author.