Multiple criteria decision analysis for therapeutic innovations in a hemophilia care center: A pilot study of the organizational impact of innovation in hemophilia care management.

BACKGROUND: Several innovative drugs liable to lead to changes in healthcare organization are or soon will be available for the management of hemophilia. Analyzing their implementation can shed further light on healthcare decision-making, to anticipate changes and risk of breakdown in the patient's care pathway.
METHODS: Multiple criteria decision analysis (MCDA), based on ISPOR recommendations, was used to assess the organizational impact of innovation in hemophilia care management. The MCDA process designed for this specific context involved ten French experts in hemophilia care management (physicians, nurses, pharmacist, physiotherapist and psychologist) in the hemophilia care center of Chambéry, in the Rhône-Alpes Region of France. This pilot study involved seven steps: (i) defining the decision problem; (ii) selecting and structuring criteria; (iii) assessing the relative weight of each criterion with software-assisted simulation based on pairwise comparisons of different organizational change scenarios; (iv) measuring the performance of the selected innovations; (v) scoring alternatives; (vi) calculating aggregate scores; (vii) discussion. The endpoint was to determine the expected overall organizational impact on a 0-100 scale.
RESULTS: Seven organizational criteria were selected. "Acceptability for patient/caregiver/association" was the most heavily weighted. Factor VIII by subcutaneous route obtained the highest aggregate score: i.e., low impact on care organization (88.8 out of 100). The innovation with strongest organizational impact was gene therapy (27.3 out of 100).
CONCLUSION: This approach provided a useful support for discussion, integrating organizational aspects in the treatment decision-making process, at healthcare team level. The study needs repeating in a few years' time and in other hemophilia centers.

Introduction

Until the turn of the 21st century, hemophilia treatment basically consisted in repeated intravenous replacement of the deficient coagulation factor [1]. For some years now, new therapies have been emerging with different, non-replacement action mechanisms, less burdensome administration route (subcutaneous), longer action duration and, in the case of gene therapy, a curative aim [2-5]. Some are real technological innovations liable to disrupt care organization and/or patients’ care pathways. Management of hemophilia, which is a rare condition, is highly structured and governed by guidelines for treatment and care organization. Patients are managed in specialized structures ensuring global multidisciplinary care. In France, the structures are hospital-based and the clotting factor concentrates are available only in hospital pharmacies. The patient’s pathway is thus highly hospital-centered; but this is expected to change, as, in France, new treatments become available in community pharmacies, closer to the patient. These less burdensome treatments and accompanying clinical improvements are liable to move care away from specialized structures and to involve new actors in patient management [6]. To maintain optimal secure care, these organizational impacts need to be studied to accompany the changes in pathway.

Introducing an innovation impacts both the organization governing it and its environment [7]. The organizational aspect is still not fully taken into account, but is emerging as one of the key aspects of the decision-making process, alongside the clinical and medico-economic aspects. Assessment of therapeutic innovations (healthcare products, acts, care organization) does not at present take account of all aspects of healthcare organization, and this can lead to irrational decisions and non-optimal use of resources. A study with 53 European hospital managers showed that organizational aspects of new treatments were considered among the most relevant information for decision-making, with safety, clinical and economic aspects [8].

This finding has led to a need for effective decision support tools to enable a systemic approach to decision-making issues [9]. According to a special task force, to identify, collect and structure information required by those making judgments to support the deliberative process, particularly in the field of health, using a multiple criterion decision analysis (MCDA) is a correct route to go down [10]. Developed in the 1970s and widely used in non-medical fields such as agriculture, energy and marketing [11-13], MCDA approaches have spread to the health field since the 2000s [14]. MCDA is especially appropriate to questions requiring a range of relevant but often heterogeneous criteria to be taken on board. It facilitates strategic and operational decision-making in fuzzy or unstable settings, identifying the decision-making problem and shedding useful light on complex issues by implementing a number of alternatives. It also has the advantage of taking account of all relevant aspects (systemic approach), in a multidisciplinary setting, stimulating exchange in a structured decision-making process and enhancing the transparency and rationality of the decisions. To cite different examples of the value of using this methodology: the Latin American Federation of the Pharmaceutical Industry (FIFARMA) stated that MCDA should strongly be considered as a tool to support Health Technology Assessment (HTA) [15]; and MCDA has been recognized as a relevant approach by the independent expert advisory committee of the Canadian Health Ministry in the context of drug reimbursement decision-making [16]. Recently, the French National Authority for Health (HAS) published, in December 2020, a methodology guide about organizational impact map for health technology assessment based on an MCDA approach [17]. In the MCDA method, the revelation of preferences can be obtained via the discrete choice approach (DCE) or via the Potentially all pairwise rankings of all possible alternatives (PAPRIKA) approach. The DCE method compares scenarios composed of four to six criteria on average [18], whereas the PAPRIKA method compares pairs of scenarios by evaluating two criteria at a time [19]. In the DCE method, the comparison of two alternatives by evaluating more than two criteria can lead to a bias if the user concentrates on one part of the criteria. The analysis conducted is then not exhaustive and can lead to uncertain results [18]. Conversely, the PAPRIKA method allows to simplify the choice of the users. Implicit pairs are automatically eliminated, which reduces the number of pairs to be compared and avoids inconsistencies in classification and the risk of redundancy [18]. This PAPRIKA method is integrated with an online software 1000minds® [20] which has the advantage of generating a random order of comparisons thus reducing the risk of bias related to the order. Moreover, the system does not involve any additional calculation and generates the global (mean or median) and individual weights for each criterion.

In the case before us, the MCDA approach has already been used in the evaluation of the organizational impacts of a therapeutic innovation, such as in acute pain management in an emergency department [9]. Despite some limitations of modeling, this demonstration provided conclusive results. To our knowledge, only one study has used MCDA in the setting of hemophilia. Gourzoulidis et al. (2021) determined’ the value of prophylaxis versus on-demand treatment in hemophilia A and emicizumab versus replacement therapy in the Greek healthcare setting [21].

Hemophilia management requires complex structured organization comprising specialized multidisciplinary teams with numerous interfaces with the various actors involved in patient management. Many parameters are relevant to treatment decision-making: type of hemophilia, severity, comorbidities, age, and the patient’s personal experience [22]. Introducing several innovations in this context is liable to induce changes in care management organization.

This context appears appropriate to develop the MCDA methodology for the assessment of the organizational impacts of a therapeutic innovation. In our view, this seems relevant because the more complex the system, the greater the uncertainty, especially in biopsychosocial models [23]. Moreover, working to reduce uncertainty will also improve the accompaniment of innovation-induced change, and make greater use of the benefits for improved health.

Thus, we conducted a pilot study in a single hemophilia care center to evaluate the organizational impact of innovation, using MCDA.

Materials and methods

The Scientific and Ethical Committee of Hospices Civils de Lyon IRB n° 00013204 (Lyon, France) examined the protocol registered under file number 21_322 and waived the requirement for ethics approval. According to French regulations and General Data Protection Regulation, all participants are informed and given the right to object. Since no medical or personal data was collected, informed oral consent was collected during the first meeting and documented in the meeting report.

The study methodology conformed to that of Lvovschi et al. [9]. The good practice guidelines of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) for the implementation of MCDA by health professionals were followed. Quantitative MCDA was used, as its methodology is well described, it is mainly used in the health field, and it has the advantage of providing aggregate scores that facilitate decision-making.

Seven steps, following the ISPOR quantitative MCDA approach, were followed: (i) defining the decision problem, (ii) selecting and structuring criteria, (iii) weighting criteria, (iv) measuring performance of alternatives, (v) generating a performance score, (vi) calculating aggregate scores, (vii) and deliberation [24, 25].

Defining the decision problem

The main study objective was to assess the impact of an innovation on the organization of care management of hemophilia patients. Secondary objectives were to define the main criteria underlying this organization and to collect the opinions of those involved regarding the interest of the MCDA approach. The analysis was performed from the point of view of health professionals specializing in hemophilia. Through this procedure organizational impact scores were produced to determine the impact of an innovation on the organization of care: weak impact, requiring minor changes, or strong impact requiring considerable changes.

A multidisciplinary expert group was set up. Inclusion criteria comprised (i) involvement in patient management in the center, (ii) familiarity with care organization, and (iii) being a medical or non-medical health professional. Individuals recruited were identified following an initial study on the care pathway of patient with hemophilia [26]. Among the 11 health professionals previously identified, only the biologist declined to participate. This led to 10 inclusions: 3 physicians, 3 nurses, 2 pharmacists, 1 physiotherapist and 1 psychologist. The study was conducted by a research team with experience in MCDA.

The study took place in the hemophilia treatment center of Chambéry in the Auvergne-Rhône-Alpes area of France, from January to June 2021, with 2 face-to-face meetings of 2 hours each and a remote individual work session (Fig 1).

Fig 1

Description of the MCDA process assessing the organizational impact of innovations.

In the first meeting, the MCDA approach and its interest for assessing the organizational impact of innovations were presented to the group, which then chose to score the organizational impact of the 6 therapeutic innovations shown in Table 1 [2, 3, 5, 27, 28].

Table 1

Alternatives selected by the expert group.

THERAPY	TYPE OF THERAPY	INNOVATIVE CHARACTERISTIC	ACCESS TO THERAPY
BIVV001	Replacement (recombinant factor VIII)	Fusion protein with XTEN polypeptides to extend half-life	Clinical trial (phase III) (hemophilia A)
OCTA101	Replacement (recombinant factor VIII)	Subcutaneous anti-hemophilia factor	Clinical trial (phase I/II)(hemophilia A)
emicizumab	Non-replacement (bispecific antibodies mimicking factor VIII action)	Action mechanism: enhancing coagulation. Long half-life. Subcutaneous route	Marketed(hemophilia A including patients with inhibitors)
concizumab	Non-replacement (anti-TFPI monoclonal antibodies)	Action mechanism: rebalancing coagulation. Long half-life. Subcutaneous route	Clinical trial (phase III)(hemophilia A and B including patients with inhibitors
fitusiran	Non-replacement (RNA interference against antithrombin)	Action mechanism: rebalancing coagulation. Long half-life. Subcutaneous route	Clinical trial (phase III)(hemophilia A and B including patients with inhibitors)
volactocogene roxaparvovecgiroctocogene fitelparvovecfidanacogene elaparvovecetranacogene dezaparvovec	Gene therapy	Healthy gene introduced in a single administration	Clinical trial (phase III)(hemophilia A and B including patients with inhibitors)

Selecting and structuring criteria

Organizational criteria were selected by the group from a list based on a review of the literature, and grouped in 5 domains: task, surrounding, peoples, structure and technology [29]. Each expert consulted the criteria for each domain to select those that seemed relevant with respect to the predetermined issue. If the proposed criteria did not seem adequate, they could add one in a “free text” field. They were asked to select at least 1 per domain and not more than 8 criteria per domain.

Criteria were validated by consensus after this individual selection stage in the first meeting. The performance levels for each criterion was then established by consensus.

Weighting criteria

Weighting used the PAPRIKA method (Potentially all pairwise rankings of all possible alternatives), a method to reveal preferences integrated in an on-line software (1000minds Ltd, Dunedin, New Zealand) [19]. The principle is based on pairwise comparison between scenarios, assessing two criteria at a time and assuming all others to be equal. Before weighting a criterion, particular attention should be paid to performance level: low, intermediate, high. For the study, each expert chose the situation they would prefer, answering the following question: “In the context of a novel hemophilia treatment, which situation would you prefer (in terms of impact on the organization of care)?” The upper limit thus corresponded to the lowest-impact situation and the highest weighting.

An explanatory notice was drawn up to optimize use of the tool. The experts provided their individual preference via the software. The mean weighting was used to calculate the aggregate score.

Measuring performance of alternatives

The second meeting reviewed the main characteristics of the innovations according to the current state of knowledge. The performance level of the alternatives was then assessed by consensus after a step of individual assessment.

Scoring alternatives and calculating aggregate scores

The performance scores and aggregate score of each innovation were calculated automatically by the PAPRIKA software after entering the performance levels validated by the expert group.

The method for aggregating weightings and performance levels most often used in the health field is the weighted sum method: where S(a) is the aggregate score, w the relative weight of the criterion, and s(a) the performance score for alternative A for criterion i.

Results

Choice and description of criteria

Eight criteria were identified as priorities in the first meeting: 2 in the “task” domain, 1 in “surrounding”, 1 in “peoples”, 1 in “structure” and 2 in “technology”.” The “structure” criterion (innovation matching network objectives) was eliminated in the assessment meeting; at care-unit level, it is rather a precondition for introducing an innovation rather than a criterion that varies with department organization depending on the innovation in question. Performance levels per criterion ranged between 3 and 4; Table 2 shows criteria and performance levels.

Table 2

Criteria for evaluation of the organizational impact.

Field	Criteria	Items	Item definitions
Task	Number of emergency or unscheduled consultations	Strong increase	Increase in number of emergency and unscheduled consultations
		Increase	Increase in number of unscheduled consultations
		No change	No change
		Decrease	Decrease in number of emergency consultations
		Strong decrease	Decrease in number of emergency and unscheduled consultations
	Change in patient recruitment	Complexification	Need for additional decision criteria
		No change	No change
		Simplification	Innovation meeting non-covered therapeutic need
Surrounding	Number of pathway or interface actors	Increase	New actors along pathway
		No change	No change
		Decrease	Fewer actors along pathway
Peoples	Medical and non-medical personnel training	Considerable	Very complex theoretical and practical training requiring development of several supports; long training (several days)
		Classical or standard	Complex theoretical and practical training requiring development of a single support accessible to all types of health professional; duration <1 day.
		Lightened	Simple theoretical and practical training requiring no new support; duration negligible (<1h)
		None	No training required
Structure	Acceptability for patient/care-giver/association	None	No contribution in terms of efficacy, tolerance, practicality, adherence
		Weak	Improves 1 criterion of efficacy, tolerance, practicality, adherence
		Moderate	Improves 1 or 2 criteria of efficacy, tolerance, practicality, adherence
		Strong	Improves several criteria of efficacy, tolerance, practicality, adherence
Technology	Supply (product, associated devices)	Very complex	Several ordering procedures for products and devices, different supply procedures and supply times
		Complex	One additional step, easy to include in current procedures
		Simple	No change
	Complexity of innovation and informed choice	Very complex	Action mechanism hard to understand even for professionals; difficult to popularize
		Complex	Action mechanism more difficult to popularize
		Simple	Action mechanism easy to explain

Weightings of criteria

The experts made between 46 and 84 explicit comparisons in a median 14 minutes (range, 9–120 min). Those who took longest (55 and 120 minutes) did not complete the analysis, due to interruption: 1 nurse (who began again), and 1 physiotherapist; the physiotherapist’s partial analysis was not taken into account in weighting the criteria.

The 3 criteria with the strongest weightings were “acceptability for patient/care-giver/association” (weight: 31.4%), “number of emergency or unscheduled consultations” (weight: 20.3%) and “complexity of the innovation and informed choice” (weight: 16.6%). Fig 2 shows criterion weightings.

Fig 2

Mean and individual weighting per criterion.

Assessment of organizational impact of innovations

Alternatives were assessed by 4 of the experts: 1 physician, 2 nurses and 1 pharmacist. Responses varied according to profession and individual factors of understanding the criteria and knowledge of the innovation. After compiling individual results, discrepancies were discussed and consensus was reached for all innovations. The “number of emergency or unscheduled consultations” criterion had the same score for all innovations (13.3 points) except for gene therapy (18.7 points). This was accounted for by the small number of emergency or non-scheduled consultations in everyday practice, leaving little room for improvement on this criterion. Treatments were presumed to be effective in the light of the state of knowledge at the time of the study. Thus, no deterioration in this criterion was expected. Responses to the “supply” criterion revealed a need for better definition, taking account of the laboratory supplying the treatment. For treatments still at the stage of clinical trials, supply modalities were difficult to determine. Obviously, the experts assessed this criterion taking account of the laboratory’s expertise: hindsight on the knowledge of the pathology and patients’ needs in terms of associated devices and the ability to set up emergency supply routes. One innovation could not be assessed on all criteria, as the experts lacked knowledge of the treatment. The 2 innovations with the highest aggregate scores (i.e., low impact on care organization) were subcutaneous recombinant anti-hemophilia factor (87/100) and very long action recombinant anti-hemophilia factor (88.8/100). Low scores (i.e., strong impact) were attributed to gene therapy (27.3/100) and anti-antithrombin RNA (21.9/68.3); their acceptability was deemed low and the innovation was considered difficult to explain in securing informed patient choice. Table 3 shows aggregate scores per innovation.

Table 3

Organizational impact scores per innovation.

	BIVV001	OCTA101	emicizumab	concizumab	fitusiran	gene therapy
Acceptability	31.4	31.4	31.4	19.3	8.6	8.6
Number of consultations	13.3	13.3	13.3	13.3	13.3	18.7
Complexity	16.6	16.6	8.1	8.1	0.0	0.0
Supply	11.7	11.7	6.1	11.7	0.0*	0.0
Training	7.7	7.7	7.7	7.7	0.0*	0.0
Recruitment	3.9	5.7	0.0	0.0	0.0*	0.0
Number of actors	2.4	2.4	0.0	0.0	0.0*	0.0
Aggregate score	87	88.8	66.6	60.1	21.9	27.3

* In the absence of sufficient data in the current state of knowledge

Discussion

This study was conducted in the field of hemophilia care. And as it took place in France, some local conditions and standards of care must be taken into account. For example, while France has implemented all 10 principles of hemophilia care (while other European countries have not), it is recognized that various aspects of the French hemophilia patient’s care need to be improved (aging, psychosocial support, etc.) [27, 30, 31]. All the more so with regard to the therapeutic innovations that are coming.

Therefore, this study shows that the use of a structured analysis method helps to formalize the essential organizational criteria in the management of a pathology and to become aware of the possible impacts in the event of a breakthrough innovation in therapies. Thus, therapeutic innovations can induce organizational innovations, and recognizing this early and precisely can make it possible to prepare adequate support for change for care teams.

Then, the MCDA method, used in this study, allowed an expert group to attribute organizational impact scores for each study innovation. The method assessed the organizational impact of innovations and identified useful points for reflection for a subsequent more qualitative analysis of the nature of such organizational changes. An innovation that is complex to explain may require setting up new explanatory tools, impacting care-unit organization. In contrast, innovations consisting in replacement showed the least organizational impact, although this is to be read in the context of the study situation and cannot be directly extrapolated to other hemophilia centers. Although hemophilia management is highly structured and coordinated, there may be organizational differences between centers according to their geographic situation and history. It would be useful to repeat the present study in other centers, to analyze variations in choice of relevant organizational criteria and possible differences in impact according to care organization and geographical region. This could complement pathway analysis, with optimization according to organizational specificities, and also a greater consideration of complementary actors, like city caregivers. For example, in France, since June 15, 2021 (after this study), the pharmaceutical product emicizumab has had a dual dispensing circuit: hospital pharmacy and city pharmacy. This is the first hemophilia treatment available to patients in private practices. This dispensing is accompanied by additional risk-reduction measures, like a new organizational chart [32], an optimized information flow and a full training on the disease and treatments.

A better understanding of the nature and strength of the organizational impact of a therapeutic innovation should allow for a better match between therapeutic choice and individual’s health determinants (biological characteristics, family environment, local health system, cultural context, etc.), to understand the most suitable patient support, and to ensure an optimal healthcare pathway, reducing the risk of dropouts.

From the point of view of the experts, the acceptability for patients/caregiver/association, the number of emergency and unscheduled consultations and the complexity of the innovation were the main criteria determining organizational impact. Gene therapy and anti-antithrombin RNA were the innovations with the greatest organizational impact, notably due to poor acceptability and high complexity hindering shared informed decision-making. Acceptability is a multifactorial criterion that may be related more to the patient and their experience and life than to the innovation itself. Thus, for example, not taking into account a good match between individual capacity for acceptability and the complexity of the innovation, which may lead to difficulties in understanding its usefulness or usability, could lead to a risk of poor therapeutic adherence. In this case, having already planned a specific work to increase the patient’s level of health literacy could be a key success factor. Poor therapeutic adherence could also result from supply difficulties inherent to a therapeutic innovation if these were not clearly envisaged beforehand.

At patient level, individual variation in organizational impact score can be expected due to the strong weighting of this criterion. In a qualitative study in 20 hemophilia patients, 40% were very favorable and 35% favorable toward gene therapy, after information delivered ahead of the interview [33]. This study underlined similarities in the organizational impact from gene therapy and anti-antithrombin RNA related to their mechanism of action. It was estimated as very complex by this expert group. The organizational impact of these two therapeutic innovations differed in terms of the number of emergency or unscheduled consultations. This criterion was considered favorable only for gene therapy because of its effectiveness in treating hemophilia. New tools or process need to be implemented in order to bring appropriate information at patient for shared-decision. This need is not specific to this team since other studies stressed the need to develop means of training or communication to facilitate shared decision-making [34-37], in agreement with the present expert group’s opinion regarding the importance of information for shared decision-making in hemophilia management. In this sense, it has already been demonstrated elsewhere that gene therapy brings a different therapeutic value chain than traditional pharma, and involves new capabilities and requirements [38]. It calls for a rethink of current collaboration: additional roles of a hemophilia treatment centers could be implemented, and ‘hub-and-spoke’ models are thought out (prescription and management in national hubs; monitoring in spokes centers) [39]. This makes it crucial to educate patient and health care personnel, essential to define new quality and safety criteria [38].

The present study contributes to the early phase of assessment of MCDA in organizational impact in the health field. The MCDA method aims to facilitate the identification of the optimal solution [40]. It is particularly appropriate for problems that require taking into account a set of relevant and often heterogeneous criteria. It facilitates strategic or operational decision-making in an imprecise or unstable environment, objectifies a decision-making problem, and analyzes complex problems by implementing several possible alternatives. This method has several advantages. It allows (i) to take into account all relevant aspects (systemic approach), (ii) to be part of a multidisciplinary approach, (iii) to stimulate exchanges within the framework of a structured decision-making process, and (iv) to improve the transparent and reasoned character of decision-making. However, the MCDA method has several drawbacks [24, 41]. First, sufficient material and human resources (time, expertise, commitment) must be available and planned. In addition, training and familiarization of professionals are prerequisites for the efficient implementation of these methods. On the other hand, the limitations of multi-criteria methods are mainly related to reliability (accurate and complete data), variability and level of evidence of the input data, and structural uncertainty, such as disagreement on the weighting method. In this regard, several limitations are to be borne in mind.

The composition of the group immediately follows the starting point of any multi-criteria method, the definition of the problem may present biases in this study. It has an impact on the choice of alternatives, criteria and performance levels that structure the problem. Moreover, the group, composed of individuals from different backgrounds, is at the origin of the weighting of the criteria and their performance levels. Variations are expected; the aggregation of individual results with or without (average value) group consensus allows participants to express their point of view. Its composition is a crucial element of the multi-criteria method [10]. It is a source of heterogeneity which must be controlled via sensitivity analysis (study of subgroups, exploration of uncertainties and disagreements with other groups) [24].

A single workshop with participants may not be sufficient to ensure a representative evaluation, and multiple workshops or surveys may be necessary [24]. In this exploratory work, two workshops were held and the individual preference revelation step was conducted remotely.

Regarding the number of experts in the group, two other studies were performed with the same method, comprising 8 and 7 experts respectively [9, 40]. However, studies need repeating to test the method in larger groups so as to resolve issues of heterogeneity and assess varied alternatives.

No patients were included in the group, although patients play a growing role in decisions affecting their health. Including them in the decision-making process is coherent with the development of pathway-based approaches and of interactions with professionals to achieve transverse dialogue. The present study will be followed up by a qualitative study in patients regarding their perception of the impact on their care organization [42]. Finally, training participants is essential, and in the present study the experts received no specific training, although the MCDA method was presented at the outset of the study.

Regarding the PAPRIKA weighting method, several authors reported that users could find it difficult to decide about certain scenarios. They find it difficult to choose between two alternatives while setting aside important criteria not involved in the proposed scenarios. The present expert group did not raise this issue. Goetghebeur et al. also reported that some users found it hard to express their own point of view rather than projecting into an institutional or social perspective. The perspective in the present study was that of the treatment center. PAPRIKA is an ordinal method, more robust than those used in grading scales for decision aid such as EVIDEM (Evidence and Value: Impact on DEcisionMaking) or Matrix4Value® [43-45]. It avoids several biases inherent to grading scales and makes comparison of weightings more reliable. The problem of individual perception of the scale is circumvented. And comparing two alternatives strictly in terms of two criteria prevents the user concentrating on only one part of the criteria.

The individual preferences expressed by the experts were seen as homogeneous by the group as a whole. However, individual weightings were closer for the criterion with the greatest weight (acceptability). Other authors likewise reported comparable weightings between assessors for the criteria judged more important; in the present study, individual weightings for criteria judged to be of intermediate importance (training and supply) were more divergent. This was also found by Goetghebeur et al. and Martelli et al.. According to Goetghebeur et al., divergences in weighting corresponded to varying individual perceptions and representations, and were not an obstacle; rather, they contribute to the exchanges generating consensus. These methods are interesting as they reveal disagreements between participants.

The same experts determined the weightings of the criteria and assessed the alternatives on these criteria. Other authors proceeded in the same way; however, Sampietro-Colom. et al. suggested that this could introduce bias. The Matrix4value® method uses two groups of experts; but this requires having sufficient human resources.

A question arises concerning maintaining non-discriminating criteria in assessing alternatives. In the present study, number of consultations was non-discriminating between the innovations with the exception of gene therapy. However, these criteria still have an impact. Increasing the number of performance levels for these criteria could confirm or discount their non-discriminating character. Moreover, in case of doubt regarding a difference in performance between several alternatives for a given criterion, the same score was attributed, and the criterion became non-discriminating, without, however, being canceled. Opinions differ on this point. Martelli et al. and Sampietro-Colom et al. attributed a score of zero in case of doubt, whereas Goetghebeur et al. attributed a low score in the EVIDEM model.

No sensitivity analysis was performed in the present study. Our MCDA model had several sources of uncertainty, affecting all components [24, 41]. Structural uncertainty was presumed to be low, as the weighting method and criterion value tree were established consensually. Uncertainty mainly came from using expert opinions as entry data. If a performance measure had been used, the choice between introducing an uncertainty criterion or an uncertainty analysis could have been discussed [24]. Oliveira et al. reported that different approaches to uncertainty analysis are used in MCDA to assess health technologies [46]. Moreover, the present low level of structural uncertainty could be confirmed by scenario analysis [24]. Finally, heterogeneity of preference could be studied by calling upon other expert groups [24].

In the absence of any prior assessment of the impact of the selected innovations on the organization of hemophilia care, no comparison could be made between a previous assessment process and the present method, as proposed by Martelli et al. and Goetghebeur et al. [43, 44]. Simulations could be set up to this end.

Conclusion

The interest of studying organizational impact is now recognized and including it in health technology assessment is a goal for the coming years. The idea of cost-effectiveness is now well understood by decision-makers. However, what choice should be made about an innovation of clear clinical efficacy but requiring changes in pathway organization and relations between actors? The distinction between medico-economic and organizational evaluation needs to be retained; but the study of organizational impact can enrich medico-economic evaluation. The question of level of evidence arises for the organizational dimension. Moreover, taking organizational impact into account complexifies the evaluation of innovations. Methods must be chosen that respect a balance between simplicity of implementation and methodological rigor. It is also important to decide at what point in the cycle of an innovation organizational impact can most usefully be assessed.

21 Jun 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Wolfgang Miesbach, MD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

"Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a potential interesting manuscript on the potential organizational impact of new therapeutic options for the treatment of hemophilia. The authors used a decision analysis based on several criteria.

Generally of interest, several points should be clarified.

The advantages and disadvantages of the MCDA method should be discussed, as well as some limitations.

What possible consequences can be drawn from the results of this study? Some haemophilia experts should comment.

Some of the requirements for gene therapy (described in the hub-and-spoke model) should be added.

Since this study was conducted in France, some of the local conditions and standard of care should be considered.

What are the organizational implications of gene therapy for hemophilia and anti-antithrombin RNA? Are there any similarities or dissimilarities?

Please correct some typos, e.g., futisuran, etc.

Reviewer #2: Thank you for your submission which makes an important contribution to our understanding of the changing treatment landscape.

There are however a couple of points that I feel need clarification and or amendment.

Line 74

You state that “to our knowledge, MCDA has never been applied to haemophilia innovations”.

This maybe correct but there have been a number of discrete choice experiments (DCE), including that of Woollacott et al (2022) that have examined professional perspectives on new haemophilia treatments. Is there a significant enough difference between an MCDA and a DCE approaches that favours MCDA in this context? I think the introduction would benefit from a short discussion on this and why you chose MCDA over DCE.

Line 85

You state that this is a pilot study. This should be reflected in both the title and in the abstract.

Line 112ff

You state how many individuals were recruited and their composition but do not mention how many were approached, declined to participate and what the composition of this group was.

Also you do not discuss whether there were any possible biases in the group. Again, some discussion of this would benefit the paper.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

8 Jul 2022

Dear reviewers,

Thank you for careful and thorough reading of this manuscript and for the thoughtful comments and constructive suggestions, which help to improve the quality of this manuscript. Our response follows (the reviewer’s comments and our responses).

Reviewer #1: This is a potential interesting manuscript on the potential organizational impact of new therapeutic options for the treatment of hemophilia. The authors used a decision analysis based on several criteria.

Generally of interest, several points should be clarified.

#1 The advantages and disadvantages of the MCDA method should be discussed, as well as some limitations.

We added line 278 :

« The MCDA method [43] aims to facilitate the identification of the optimal solution. It is particularly appropriate for problems that require taking into account a set of relevant and often heterogeneous criteria. It facilitates strategic or operational decision-making in an imprecise or unstable environment, objectifies a decision-making problem, and analyzes complex problems by implementing several possible alternatives. This method has several advantages. It allows (i) to take into account all relevant aspects (systemic approach), (ii) to be part of a multidisciplinary approach, (iii) to stimulate exchanges within the framework of a structured decision-making process, and (iv) to improve the transparent and reasoned character of decision-making. However, the MCDA method has several drawbacks. First, sufficient material and human resources (time, expertise, commitment) must be available and planned. In addition, training and familiarization of professionals are prerequisites for the efficient implementation of these methods. On the other hand, the limitations of multi-criteria methods are mainly related to reliability (accurate and complete data), variability and level of evidence of the input data, and structural uncertainty, such as disagreement on the weighting method. In this regard, several limitations are to be borne in mind.”

#2 What possible consequences can be drawn from the results of this study? Some haemophilia experts should comment.

We added some comments in the discussion section. See lines 221-225, 238-247 and 253-258.

#3 Some of the requirements for gene therapy (described in the hub-and-spoke model) should be added.

We added line 271-277 :

“In this sense, it has already been demonstrated elsewhere that gene therapy brings a different therapeutic value chain than traditional pharma, and involves new capabilities and requirements [39]. It calls for a rethink of current collaboration: additional roles of a hemophilia treatment centers could be implemented, and ‘hub-and-spoke’ models are thought out [40] (prescription and management in national hubs; monitoring in spokes centers) [41]. This makes it crucial to educate patient and health care personnel, essential to define new quality and safety criteria [42].”

#4 Since this study was conducted in France, some of the local conditions and standard of care should be considered.

We added some information in the discussion section. See lines 216-225, 238-247 and 253-258.

#5 What are the organizational implications of gene therapy for hemophilia and anti-antithrombin RNA? Are there any similarities or dissimilarities?

We added line 262: “This study underlined similarities in the organizational impact from gene therapy and anti-antithrombin RNA related to their mechanism of action. It was estimated as very complex by this expert group. The organizational impact of these two therapeutic innovations differed in terms of the number of emergency or unscheduled consultations. This criterion was considered favorable only for gene therapy because of its effectiveness in treating hemophilia. New tools or process need to be implemented in order to bring appropriate information at patient for shared-decision. This need is not specific to this team since other studies stressed the need to develop means of training or communication to facilitate shared decision-making [36-38], in agreement with the present expert group’s opinion regarding the importance of information for shared decision-making in hemophilia management.”

#6 Please correct some typos, e.g., futisuran, etc.

Typos have been corrected.

Reviewer #2: Thank you for your submission which makes an important contribution to our understanding of the changing treatment landscape.

There are however a couple of points that I feel need clarification and or amendment.

#6 Line 74 You state that “to our knowledge, MCDA has never been applied to haemophilia innovations”.

Indeed, when we submitted the article, no MCDA study analyzing hemophilia was published to our knowledge. However, we conducted a new literature search that revealed a study published in December 2021 (after our submission). Therefore, we have replaced the sentence line 84 with: "To our knowledge, only one study has used MCDA in the setting of hemophilia. Gourzoulidis et al (2021) determined' the value of prophylaxis versus on-demand treatment in hemophilia A and emicizumab versus replacement therapy in the Greek healthcare setting.”

This maybe correct but there have been a number of discrete choice experiments (DCE), including that of Woollacott et al (2022) that have examined professional perspectives on new haemophilia treatments. Is there a significant enough difference between an MCDA and a DCE approaches that favours MCDA in this context? I think the introduction would benefit from a short discussion on this and why you chose MCDA over DCE.

We added line 69 : « In the MCDA method, the revelation of preferences can be obtained via the discrete choice approach (DCE) or via the Potentially all pairwise rankings of all possible alternatives (PAPRIKA) approach. The DCE method compares scenarios composed of four to six criteria on average [18], whereas the PAPRIKA method compares pairs of scenarios by evaluating two criteria at a time [19]. In the DCE method, the comparison of two alternatives by evaluating more than two criteria can lead to a bias if the user concentrates on one part of the criteria. The analysis conducted is then not exhaustive and can lead to uncertain results [20]. Conversely, the PAPRIKA method allows to simplify the choice of the users. Implicit pairs are automatically eliminated, which reduces the number of pairs to be compared and avoids inconsistencies in classification and the risk of redundancy [20]. This PAPRIKA method is integrated with an online software 1000minds® [21] which has the advantage of generating a random order of comparisons thus reducing the risk of bias related to the order. Moreover, the system does not involve any additional calculation and generates the global (mean or median) and individual weights for each criterion [22].

#7 Line 85 You state that this is a pilot study. This should be reflected in both the title and in the abstract.

We completed the title and the abstract as recommended.

#8 Line 112 You state how many individuals were recruited and their composition but do not mention how many were approached, declined to participate and what the composition of this group was.

We added line 127: “Individuals recruited were identified following an initial study on the care pathway of patient with hemophilia [27]. Among the 11 health professionals previously identified, only the biologist declined to participate.”

#9 Also you do not discuss whether there were any possible biases in the group. Again, some discussion of this would benefit the paper.

We added line 293: « The composition of the group immediately follows the starting point of any multi-criteria method, the definition of the problem may present biases in this study. It has an impact on the choice of alternatives, criteria and performance levels that structure the problem. Moreover, the group, composed of individuals from different backgrounds, is at the origin of the weighting of the criteria and their performance levels. Variations are expected; the aggregation of individual results with or without (average value) group consensus allows participants to express their point of view. Its composition is a crucial element of the multi-criteria method [43]. It is a source of heterogeneity which must be controlled via sensitivity analysis (study of subgroups, exploration of uncertainties and disagreements with other groups) [44].

A single workshop with participants may not be sufficient to ensure a representative evaluation, and multiple workshops or surveys may be necessary. In this exploratory work, two workshops were held and the individual preference revelation step was conducted remotely [44].»

Submitted filename: Beny et al 2022 review 1 answer.pdf

Click here for additional data file.

16 Aug 2022

Multiple criteria decision analysis for therapeutic innovations in a hemophilia care center: a pilot study of the organizational impact of innovation in hemophilia care management.

PONE-D-21-37437R1

Dear Dr. Beny,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Wolfgang Miesbach, MD

Academic Editor

PLOS ONE

1 Sep 2022

PONE-D-21-37437R1

Multiple criteria decision analysis for therapeutic innovations in a hemophilia care center: a pilot study of the organizational impact of innovation in hemophilia care management.

Dear Dr. Beny:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Wolfgang Miesbach

Academic Editor

PLOS ONE