Pharmacogenomics decision support in the U-PGx project: Results and advice from clinical implementation across seven European countries.

BACKGROUND: The clinical implementation of pharmacogenomics (PGx) could be one of the first milestones towards realizing personalized medicine in routine care. However, its widespread adoption requires the availability of suitable clinical decision support (CDS) systems, which is often impeded by the fragmentation or absence of adequate health IT infrastructures. We report results of CDS implementation in the large-scale European research project Ubiquitous Pharmacogenomics (U-PGx), in which PGx CDS was rolled out and evaluated across more than 15 clinical sites in the Netherlands, Spain, Slovenia, Italy, Greece, United Kingdom and Austria, covering a wide variety of healthcare settings.
METHODS: We evaluated the CDS implementation process through qualitative and quantitative process indicators. Quantitative indicators included statistics on generated PGx reports, median time from sampled upload until report delivery and statistics on report retrievals via the mobile-based CDS tool. Adoption of different CDS tools, uptake and usability were further investigated through a user survey among healthcare providers. Results of a risk assessment conducted prior to the implementation process were retrospectively analyzed and compared to actual encountered difficulties and their impact.
RESULTS: As of March 2021, personalized PGx reports were produced from 6884 genotyped samples with a median delivery time of twenty minutes. Out of 131 invited healthcare providers, 65 completed the questionnaire (response rate: 49.6%). Overall satisfaction rates with the different CDS tools varied between 63.6% and 85.2% per tool. Delays in implementation were caused by challenges including institutional factors and complexities in the development of required tools and reference data resources, such as genotype-phenotype mappings.
CONCLUSIONS: We demonstrated the feasibility of implementing a standardized PGx decision support solution in a multinational, multi-language and multi-center setting. Remaining challenges for future wide-scale roll-out include the harmonization of existing PGx information in guidelines and drug labels, the need for strategies to lower the barrier of PGx CDS adoption for healthcare institutions and providers, and easier compliance with regulatory and legal frameworks.

Background

With the availability of guidelines that provide evidence-based therapeutic recommendations for more than 100 gene-drug pairs, clinical pharmacogenomics (PGx) has the potential to be one of the first milestones on the way to realize personalized medicine in routine clinical care [1-3]. Ever since the first PGx implementation programs were devised, clinical decision support (CDS) systems have proven to be indispensable components of PGx implementation by distilling the vast amount of information contained within a patient’s PGx results into clinically actionable and concise therapeutic recommendations [4-8].

While the implementation of PGx-guided therapy has gained some traction in certain specialized early adopter sites, particularly in the US, within the past two decades, broader adoption has not yet occurred [5–7, 9]. General challenges to incorporation of PGx into clinical practice include lack of awareness and education about PGx among healthcare providers, lack of data on cost-effectiveness of multi-panel testing approaches and consequently reimbursement issues [10]. These challenges are further aggravated by the limitations and diversity of the current European infrastructural healthcare landscapes in terms of availability, capability and interconnectedness of Electronic Health Record Systems (EHRs), which form the prerequisite for the most efficient delivery of PGx CDS, i.e., through the display of automatic alerts during the prescribing process [11-14].

To address these challenges, the Ubiquitous Pharmacogenomics (U-PGx) consortium was initiated in 2016 with three primary aims: First, to address current barriers for PGx implementation including the provision of enabling tools. Second, to yield scientific evidence for the improvement of patient outcomes through pre-emptive PGx testing by means of a prospective, randomized clinical study. And third, to evaluate the cost-effectiveness of such an approach [15, 16].

The project implements a so-called ‘pre-emptive’ PGx testing approach, where an individual is tested for a panel of multiple genes at once instead of conducting multiple single gene tests over time, facilitating PGx-guided treatment for a number of gene-drug combinations throughout an individual’s life [5, 17–19].

To enable the clinical implementation of pre-emptive PGx testing across all seven participating European countries, despite large diversity in infrastructural conditions, the U-PGx consortium has conceptualized a unique multi-modal, multi-language PGx CDS framework that offers a standardized implementation framework while also meeting the individual technical requirements and capabilities of each clinical site [14].

The effectiveness of CDS hinges on its adoption by healthcare providers. Previous research has reported mixed results regarding satisfaction with and adoption of PGx CDS [20-24]. Further, the majority of previous PGx CDS evaluation studies was conducted in simulated study settings rather than in real-world clinical settings [19, 21, 23–27].

In contrast, this study investigated the implementation and usability of a multi-modal CDS infrastructure in a real-world, multinational clinical setting. Further, to the best of our knowledge, this is the first study that examines a CDS infrastructure that consists of multiple, complementary tools conceptualized to accommodate a variety of health IT infrastructures and capabilities. In particular, we also investigated the acceptance and usefulness of a mobile-based CDS tool that was designed as an auxiliary tool to enable using and sharing of PGx test results in healthcare settings where the EHR infrastructure is fragmented or missing altogether.

To evaluate the implementation process and the usability of the provided CDS tools, we used a triangulated research approach combining quantitative and qualitative data including a user study.

In the following, we describe process indicators on uptake and usability, an analysis of qualitative feedback on workflow efficiency, bottlenecks and optimization potential of the U-PGx implementation framework, and share insights and experiences from all phases and layers of the implementation process. Furthermore, we provide considerations on prerequisites and challenges for continuation and expansion of the PGx CDS implementation in Europe, including practical advice that can be of value for similar projects. For reporting our implementation strategy, we follow the Standards for Reporting Implementation Studies (StaRI) checklist (see S5 File) [28].

Methods

Setting

The U-PGx project is a large-scale, multinational European implementation project to investigate the impact of pre-emptive PGx testing on patient outcomes in a clinical study (“PREPARE”, ClinicalTrials.gov Identifier: NCT03093818), and the cost-effectiveness of such an approach [15].

PREPARE was designed as a prospective, block-randomized, controlled study. More than 6,800 patients were enrolled over the course of three years at one or more clinical sites across the seven participating countries. The participating countries were randomized to start with either PGx-guided treatment or with ‘standard of care’ (control arm) in the first block of 18 months. After 18 months, the countries switched to the opposite arm and recruited new patients for another 18 months. The primary outcome measure was the occurrence of at least one clinically relevant adverse drug reaction caused by the drug of inclusion. Additional details on the study design can be found in [15].

More than 15 clinical sites across seven participating European countries (i.e., the Netherlands, Spain, Slovenia, Italy, Greece, United Kingdom and Austria) with varying health IT infrastructures—ranging from complete absence of any such infrastructure to the capability to display CDS alerts via the electronic health record—are participating in the project.

To address the complex requirements of the U-PGx project setup, a unique assortment of complementary multi-modal decision support tools was developed around a central knowledge management platform and successively implemented from 2017.

Pseudonymized raw results from the genotyping platform are automatically or manually transferred to the secure central platform, and patient-specific results and recommendations based on the guidelines authored by the Dutch Pharmacogenetics Working Group of the Royal Dutch Pharmacists Association for more than 40 drugs associated with one or more of 13 different genes can be retrieved in various formats and in seven different languages.

Depending on their local infrastructure and preferences, clinical sites chose to use one or more of the following CDS tools: 1) Semi-structured data that can be integrated with existing local EHRs and used to display automatic alerts in the prescribing process, 2) a conventional report format that can be printed or filed in the EHR, and 3) a credit-card sized “Safety Code” card (www.safety-code.org) that is handed to the patient and enables on-the-fly access of patient-specific dosing recommendations via a smartphone or tablet using QR codes (Figs 1 and 2). The technical set-up and implementation of these tools, and the IT infrastructures and capabilities of each participating clinical site have been described in detail previously [14].

Fig 1

Framework of complementary decision support solutions implemented in the U-PGx project.

The provided set of solutions accommodates the needs and requirements of the highly diverse implementation sides while guaranteeing a standardized intervention. Clinical sites with the infrastructural capabilities for interruptive CDS alerts in the form of pop-up messages in the EHR or e-prescription system are provided with semi-structured data. For all other sites, passive CDS can be delivered either inside the EHR system as a digital report, or outside the EHR system via mobile- and paper-based solutions. A more detailed technical description is available in [14].

Fig 2

Front and back side of an exemplary Safety Code card for a fictional patient.

The Safety Code card is given to all patients recruited for the ‘PGx-guided treatment’ arm of the study. Patients recruited for the control arm were eligible to receive a Safety Code card at the end of the study [14].

Pre-implementation phase

Fig 3 shows the different phases of the implementation process including the tools and methods used for monitoring and evaluation. The different phases including evaluation methods are described in detail in the following sections.

Fig 3

Phases and tools of the implementation process.

Requirements analysis

One of the main challenges of a large-scale implementation project covering many diverse institutions and geographic regions is to provide a solution that is standardized yet flexible enough to accommodate different existing frameworks and conditions. Requirements at the different sites were assessed in an iterative process.

To gain a first overview of the current infrastructure and workflows in place at the participating institutions, a structured questionnaire was developed together with the coordinators and collaborators involved in developing and implementing the IT solutions. Covered topics included: 1) if, available, the current workflow for selecting and testing patients, 2) the storage of test results and interpreted PGx data, and 3) the local EHR infrastructure and capabilities. The complete requirements analysis template and results are provided in S1 File.

After the initial assessment, requirements of each implementation site were further specified individually. Additionally, an analysis of legal and regulatory conditions relevant to the implementation was conducted (see S2 File).

Risk assessment

A list of potential risks was compiled based on challenges and reasons for delay reported by other decision support implementation projects, the information collected in the requirements analysis and the analysis of European regulatory frameworks [4]. Based on these sources, a questionnaire was drafted, in which both clinical collaborators and all partners involved in the implementation were asked to rate the listed risks in terms of 1) expected probability (low, medium, high) and 2) expected impact (minor, medium, severe) of the respective risk (see S3 File).

The severity of identified risks was estimated based on their impact and probability level using a 3x3 risk assessment matrix. Risks with high impact and medium or high probability were rated as “severe”. Risks with low probability and high impact, and risks with medium impact and medium or high severity were rated as “medium” [29, 30].

Post-implementation phase

Evaluation framework

To monitor and gain a comprehensive view of the uptake and acceptance of the decision support tools, various quantitative and qualitative data collection methods were combined and triangulated. These methods included 1) tracking of general process indicators, 2) a standardized questionnaire targeted at the clinicians and pharmacists involved in the recruiting or treatment of patients enrolled in the PREPARE study, 3) statistics on report retrievals via the Safety Code card component of the decision support tools, and 4) qualitative feedback by the clinical study coordinators.

General process indicators

The centralized processing of samples and generation of PGx reports using the genetic information management system (GIMS) allowed the tracking of process indicators across all sites (see Fig 1). These included the number of uploaded samples and generated reports per implementation site, the turnaround time between sample upload and report delivery and genotype/phenotype frequencies. Data on process indicators was extracted from GIMS, and descriptive statistics were calculated.

Due to the design of the clinical study as a randomized crossover study, roll-out of the CDS infrastructure was scheduled in two phases, with the first group of implementation sites being scheduled for the beginning of 2017, and the second group of sites for July 2018. Data on planned roll-out dates vs. actual roll-out dates of the CDS infrastructure including reasons for delays were collected for each implementation site.

Survey

Informed by previous PGx CDS user studies, a questionnaire to evaluate the decision support tools was developed combining quantitative and qualitative question formats [21, 25]. Queried information included (1) demographic data, (2) the self-reported extent of adoption of PGx testing and the various CDS tools, and (3) the usability of the various CDS tools in terms of their perceived information content, usefulness and workflow integrability.

The questionnaire was implemented as a web-based survey using the online survey platform “Alchemer” and reviewed and tested within the consortium [31]. The questionnaire is available in S4 File.

Survey participants were identified through communication with the principal clinical investigators at the respective study sites and invited via email to participate in the online web-based survey. Response status was tracked and three reminder invitations were sent in case of no response.

Data analysis was conducted using the Python programming language and descriptive statistics were calculated using the Python NumPy and pandas libraries [32, 33]. Plots to visualize Likert items were created using Python matplotlib [34].

Statistics on report retrievals

The utilization of the Safety Code card component of the clinical decision support tools was evaluated by collecting de-identified statistics on page retrievals via the card’s QR code. A dashboard to monitor results was created using the Python programming language.

Qualitative feedback by implementation site coordinators

In addition to the user survey and quantitative process indicators, the workflow and implementation were qualitatively evaluated by surveying the clinical implementation site coordinators on the implemented workflow, encountered workflow bottlenecks (i.e., components or processes that negatively impact report delivery time), their satisfaction with the different components of the decision support infrastructure and their interest in and prerequisites for continuing to use the different components.

Ethics approval and consent to participate

This study received ethics approval from the ethics committee of the Medical University of Vienna (No. 2091/2016). All participants provided informed consent. Furthermore, the study was approved by ethics review boards in all participating countries and all necessary permits and approvals were obtained. The review boards in other participating countries were: 1) Comité Coordinador de Ética de la Investigación Biomédica de Andalucía, Sevilla (Spain), 2) Scientific Council of the Committee on research and ethics, University General Patras Hospital (Greece), 3) Comitato etico unico regionale, Istituto di ricovero e cura a carattere scientifico, Aviano (Italy), 4) Medisch Ethische Toetsingscommissie Leiden Den Haag Delft (Netherlands), 5) Komisiji za medicinsko etiko, University of Ljubljana (Slovenia), 6) NHS Health Research Authority, North West—Liverpool Central Research Ethics Committee (UK).

Results

Risk assessment

Overall, 27 risks were identified of which seven (25.9%) were rated as severe (i.e., having a severe impact and having medium or high probability). 12 (44.4%) risks were rated as medium and 8 (29.6%) as minor. Out of the seven risks rated as severe, two were related to team coordination and communication while four were related to IT and lab infrastructure issues (see Table 1).The complete risk assessment results are available in S3 File.

Table 1

Identified risks per thematic category and severity rating.

LIMS: Laboratory information management system.

Category	Severe (avoid, control, transfer)	Medium (avoid, control, transfer)	Minor (watch, assume)
Team coordination and communication	Collaboration and availability of different suppliers participating in the project	Close collaboration between ICT project partners and local sites and ICT departments required (n = 2)	N/A
	Lack of prioritization of essential tasks due to institutional factors	Financial compensation may be required for collaboration with local ICT departments
IT / Lab infrastructure	Requirement of approval of local institutional committees	Required to remotely work with a large number of differing local IT infrastructures, LIMS and EHRs and their interfaces (n = 2)	Laboratory that will provide the PGx results will move to another building during the project
	Local requirements related to data format (e.g., XML)	Lack of EHR/LIMS in some countries	Quality and consistency of translations of guidelines to local languages need to be guaranteed
	Highly diverse requirements for interfaces that may require customized solutions	Generation of paper-based reports required in addition to solutions for EHR systems
	Difficulties in implementing rules for decision support	Infrastructure and workflow for printing safety-code card needs to be devised
		Methods for de-identification required
		Training in ICT-related topics required for clinical partners
Staffing	-	Problems with hiring personnel due to bureaucratic factors (n = 2)	Difficulties in getting staff with appropriate expertise (n = 2)
		Extensive training required for new staff
Changes at implementation site	N/A	N/A	Moving between hospital buildings will occur during the project (n = 2)
			Additional departments will likely be required to be involved in the project
Other problems	Resulting report based on DPWG guidelines too long or too unstructured to be usable by clinicians	N/A	N/A
Total number	7	13	7

Risks assigned the highest severity rating were collaboration and communication challenges resulting from the large-scale, multidisciplinary setting of the project, establishing collaboration including obtaining necessary permissions from local IT departments, risks associated with establishing the data basis for the CDS solutions and devising a workflow and infrastructure suitable for each clinical site.

This structured, collaborative assessment of risks was then used to categorize risks into different action categories, i.e., ‘avoid/control/transfer’ and ‘watch/assume’ and to devise mitigation strategies, where possible. In retrospect, risks that emerged as the greatest challenges were institutional, collaborative and bureaucratic factors, such as dependency on collaborators and suppliers outside of the project for whom the project was not a priority, and the development of the required tools and reference data resources (see Section ‘General implementation process indicators’).

General implementation process indicators

For the first group, roll-out was finalized in June 2017, and for the second group in October 2018, resulting in postponements of six and three months, respectively. In the first case, delays were primarily caused by challenges including institutional factors and development of the required tools and reference data resources, such as the translation tables for conversion of variants to haplotypes and genotype-predicted phenotypes; the latter entailed many complexities due to missing standardized nomenclatures. For the second group, the recorded delay was attributable to the postponement of the first phase.

As of March 2021, PGx reports and Safety Code cards were created for 6884 genotyped samples in U-PGx GIMS. Processing of samples of control arm patients is still ongoing at the time of writing. Fig 4 shows the number of uploaded samples per country. For 94.5% of samples, a full report covering all SNPs of the U-PGx genotyping panel could be delivered. For the remaining samples, a partial report was delivered due to missing results for one or more SNPs.

Fig 4

Number of genotyping samples uploaded per country as of March 2021.

Processing of samples of control arm patients was still ongoing at the time of writing.

As of March 2021, 83.8% of uploaded samples had at least one actionable phenotype, i.e., a phenotype for which the DPWG guidelines recommended an adjustment of treatment. The median number of actionable phenotypes per sample was one. Median waiting time for report generation after sample upload was twenty minutes.

Decision support infrastructure evaluation survey

Demographics

In total, 131 healthcare providers involved in recruiting and treating patients enrolled in the PREPARE study were invited to participate in the survey. Out of these, 65 completed the questionnaire (Response rate 49,6%). Fig 5 shows the number of respondents vs. the number of invited healthcare providers per country and the distribution of respondents across countries.

Fig 5

a) Invited healthcare providers vs. number of respondents per country, b) Percentage of respondents per country.

Forty percent of participants in our sample identified as male, 58.5% as female and one participant preferred not to state their gender. This ratio corresponded approximately to the ratio among the total number of invited healthcare providers (59.5% female). Participants had a median age of 40 years, and the median number of years of working experience was twelve. The median number of years of experience with PGx was two (see Table 2).

Table 2

Demographics of the survey participants.

	Median	Q1	Q3	Max	Min
Age	40	35	50	62	22
Years of working experience	12.0	8	20	36.0	0.5
Years of experience with PGx	2.0	0.5	2	12.0	0.1
Number of PGx tests ordered per week	1	0	1	9.0	0.0

The majority of respondents (61.5%) stated to be mostly treating patients in an outpatient setting. Pharmacy (30.8%), primary care / family medicine (15.4%) and psychiatry (16.9%) were the three most often reported fields of work, followed by oncology (23.1%), cardiology (9.2%) and nephrology (4.6%). Corresponding percentages for all invited users based on the type of healthcare facilities participants were recruited from were pharmacy (18.3%), primary care / family medicine (27.5%), psychiatry (8.4%), oncology (29.0%), cardiology (11.5%) and nephrology (5.3%).

The majority of participants agreed or strongly agreed to be comfortable using computers, mobile devices and EHRs (Fig 6). Out of these three devices, EHRs and computers had the highest number of participants stating to be not comfortable using them (both n = 5, 7.7%) or being neutral towards using them (n = 3, 4.6% and n = 10, 15.4%, respectively). Respondents who indicated unfamiliarity or being uncomfortable with one or more of these devices were from Slovenia or Greece, i.e. countries with no or only partially available EHR infrastructure.

Fig 6

Healthcare provider’s self-reported comfortableness with computers, mobile devices and electronic health records (EHRs).

When asked whether they had received enough training in PGx, more than two thirds of respondents agreed (n = 39, 60%) or strongly agreed (n = 6, 9.2%), while 17 (26.2%) participants felt undecided and three felt they had received too little training.

Uptake and usage

Table 3 shows the number of users and reported median times of use per week for the different CDS tools.

Table 3

Reported usage of the different CDS tools among survey participants.

More than one CDS tool can be used by a single user.

	Number of users	Median number of uses per week	Max number of uses per week	Min number of uses per week
Automatic alerts	15	1	5	0
Digital reports	27	2	10	0
Paper-based reports	50	1	30	0
Safety Code cards	22	1	5	0
Other methods	4	1	1	1

Most participants had at least once used the paper-based reports (76.9%) or digital reports (41.5%). 23% of respondents had used automatic alerts via the EHR and 33.8% of respondents indicated to have at least once used the Safety Code card.

In four cases, participants reported to have used other methods to interpret a patient’s PGx results than the provided CDS tools. These included direct communication or communication via phone calls, e-mail or WhatsApp messages with pharmacists.

Usability evaluation

Across all tools, the majority of users felt that they integrate well with their work routine (Fig 7). The paper-based reports had the highest number of users who felt neutral about this (n = 11), followed by the digital reports (n = 5) and the Safety Code card (n = 5). For these tools, only few users indicated that they do not fit into their work routine (n = 5 in total).

Fig 7

Healthcare providers’ rating of workflow integration of the different CDS tools.

SC: Safety Code cards, PR: printed reports, DR: digital reports, AA: automatic alerts.

Similarly, across all tools, the majority of users felt that the tools provided the right amount of information (Fig 8). However, for all tools, there were also single users who either felt that too little or too much information was provided. Feeling that too much information is provided was most salient for the Spanish implementation site, where a user also pointed out a preference for their own institutional PGx reports over the ones delivered as part of the project. Free text comments about the CDS information provided indicated the wish for more concrete information for whether action needs to be taken and the wish for more practical recommendations. Many users felt that both the paper-based reports and the digital reports were generally too long, should include less information and could be improved by better structuring, which was especially salient for the paper-based reports.

Fig 8

Healthcare providers’ perception of the information amount of the different CDS tools.

SC: Safety Code cards, PR: printed reports, DR: digital reports, AA: automatic alerts.

Most respondents agreed or strongly agreed to have received enough training to confidently use the different CDS tools (Fig 9). Again, several users, mostly from Greece and the Netherlands, also felt undecided and few felt they had received too little training.

Fig 9

Healthcare providers’ perceived sufficiency of the received training on the different CDS tools.

SC: Safety Code cards, PR: printed reports, DR: digital reports, AA: automatic alerts.

For all tools, the user-friendliness was rated as “good” or “excellent” by the majority of users (Fig 10). Few users, most pronounced for the Spanish implementation site, rated the user friendliness as “Poor” with the paper-based reports having the highest number of this rating. The other categories “Worst imaginable”, “Awful” and “Best imaginable” were not chosen by any respondents.

Fig 10

Healthcare providers’ rating of the user-friendliness of the different CDS tools.

SC: Safety Code cards, PR: printed reports, DR: digital reports, AA: automatic alerts.

For all tools, the majority of users stated to be either “satisfied” or “very satisfied” with them (Fig 11). The paper-based reports and Safety Code card had the highest number of users who felt neither satisfied nor dissatisfied with them. Only two users indicated to be dissatisfied or very dissatisfied with a tool (i.e. the digital report and the Safety Code card). In the free text comments, the Safety Code card, however, received several positive comments. One user stated:

Fig 11

Healthcare providers’ overall satisfaction with the different CDS tools.

SC: Safety Code cards, PR: printed reports, DR: digital reports, AA: automatic alerts.

“A useful tool for which a caregiver can assume patients need to have it with [them] "all the time" Limited information can be printed, obviously. But a useful code (QR) can help to get more information about [the] genetic profile”

Report retrievals via the Safety Code card

In total, 4,040 report retrievals via the Safety Code card were logged from April 2018 until the end of patient recruitment on June 30 2020 (see Fig 12). Logging of report retrievals started in April 2018, thus retrievals before this date are not included. Since the end of recruitment until the time of writing (beginning of March 2021), another 1,282 report retrievals were logged. With regard to this, it is important to note that patients recruited for the control arm only received their Safety Code cards after completion of the study.

Fig 12

a) Report retrievals from April 2018 to March 2021. Report retrieval logging started in April 2018, thus retrievals before this date are not included. b) Samples uploaded vs. number of report retrievals via the Safety Code card during and after the study per implementation site. Patients recruited for the control arm received their Safety Code cards only after completion of the study.

In relation to the number of recruited patients, the Netherlands and Austria had the highest number of report retrievals, while Italy and Greece had the lowest. Returning visits (i.e. the same user retrieving more than one report) made up 61.6% of all report retrievals. Fig 12 shows the distribution of report retrievals over time.

Workflow analysis including qualitative feedback by implementation site coordinators

Overall, the prepared workflow ranging from identification of patients for PGx testing to result retrieval was followed by all implementation sites with only minor divergences, and was generally deemed efficient by clinical coordinators. Exceptions included unexpected longer time spans from sample retrieval to report submission caused by issues concerning the genotyping platform that were experienced by several sites. Suggestions for improvement included a more transparent and user-friendly workflow for identifying, translating and annotating the updated PGx recommendations in the GIMS platform.

Both Spanish and Greek implementation sites reported a low usage of the Safety Code card, because of low acceptance from the patients’ side, the healthcare providers’ side or both.

All sites expressed interest in continuing several or all components of the CDS infrastructure in the future provided a continued updating of the guidelines and maintenance of the related CDS tools.

Discussion

In this paper, we presented the results of the first European large-scale, multi-center, multi-national implementation of PGx decision support using a triangulated evaluation strategy. We combined quantitative and qualitative process indicators with qualitative meta-feedback to gain a comprehensive picture of the overall success of our implementation strategy and to identify areas for improvement, as well as remaining challenges to an extended large-scale roll-out of PGx decision support in Europe.

Evaluation of the implementation

Overall, the acceptance and satisfaction with the different CDS tools was rated positively across all investigated aspects (i.e. workflow integration, user-friendliness, information content, overall satisfaction) by the majority of involved healthcare providers. The majority of users further indicated that they felt they had both received enough training to confidently use the CDS tools and to use PGx results in general. In contrast, previous CDS implementation projects have reported mixed results on the satisfaction with PGx CDS and general training in PGx [20–24, 35–37].

There are several possible explanations for the largely positive evaluation results of the U-PGx CDS infrastructure. Most importantly, the U-PGx CDS implementation strategy differentiates from previous projects in that it offers implementation sites a spectrum of tools that can be chosen and combined based on infrastructural requirements and subjective preference. Moreover, the implementation was accompanied by a dedicated education program, offering workshops and an e-learning program to further educate and familiarize participating healthcare providers with all aspects of PGx. This education program was informed by a survey that assessed healthcare provider’s current knowledge about PGx at the beginning of the project [36, 38]. Further, a ‘train-the-trainer’ concept was used to increase the range of educational efforts, which means that project coordinators at each implementation site were trained and instructed to in turn train on-site personnel. Finally, it has to be noted that our sample of healthcare providers had, in median, two years of experience with at least some aspect of PGx, which may have made it easier for them to use the CDS tools.

While these user study results indicate an overall successful implementation, especially the opinions of less satisfied users may help to identify leverage points to increase overall acceptance among healthcare providers and future extended implementations outside the setting of a clinical study.

Areas for improvement that emerged from our evaluation included the structure and formatting of reports, and the phrasing and content of recommendations. It is well known from previous research that healthcare providers prefer PGx CDS recommendations to be concise with additional information (including links to brief PGx educational pieces) being available on request to avoid information overload [21, 23, 25, 39]. While we aimed to consider these best practices when designing the structure of reports, several constraints made this more difficult in our setting. Firstly, we used the original DPWG guideline recommendation texts including their existing data structure, which are tailored to the Dutch setting both in terms of CDS capabilities and in terms of adaptation to the local healthcare system. Secondly, our setting made it necessary to also provide paper-based reports, where the paradigm to provide more information only on request is harder to achieve, as compared to e.g., providing a link to further information in digital CDS tools. Our solution concept was to provide the most relevant information at the top of the report, while including additional information on the following report pages. However, this might still lead to a perceived information overload for some healthcare providers. Improving the design and formatting may, however, help to alleviate this issue.

One rationale of our multi-modal CDS design was to provide a solution for making PGx results and recommendations available and transferable without the need for an EHR infrastructure or the interoperability and interconnectedness between different EHR infrastructures. To achieve this, the Safety Code card was implemented at all sites. The Safety Code card enables report retrieval through standard mobile phones based on a QR-code printed together with a short summary of the PGx profile on a credit-card sized card. While our analysis revealed that the Safety Code card was accepted and considered a beneficial tool among the majority of participating health care providers, one important insight was the fact that this assessment and especially the card’s usage seemed to differ between implementation sites. For example, our evaluation showed that the Greek implementation sites, where no EHR infrastructure exists at all, had, overall, relatively lower usage of the card. Similarly, the Spanish and Italian implementation sites showed, compared to other sites and in relation to the number of recruited patients, lower usage of the Safety Code card.

Non-usage may be attributed to low acceptance from patient-side (i.e., patients not carrying their cards or not showing them to healthcare providers), low acceptance from healthcare provider-side or lower number of occasions for card utilization (e.g., lower number of future PGx drug prescriptions). Feedback from clinical coordinators pointed to low acceptance from the healthcare provider-side due to lower digital literacy at the Greek implementation site, and low acceptance from patient-side at the Spanish implementation site. Further, differences in patient population, such as median age and health status might have led to different adoption rates in different countries. For example, patients recruited at the Italian implementation sites were mostly oncological patients with a relatively higher median age. It has to be noted that familiarity with and usage of information and communication technology show large variations across European countries, and differ per age group, which may have also impacted adoption and acceptance [40]. Additionally, we found that harmonization with pre-existing solutions should be given high priority to increase acceptance among healthcare providers.

One inherent limitation of our evaluation survey is potential non-responder bias, i.e., the opinions and views of healthcare providers who chose not to participate in the survey may not be represented adequately. These healthcare providers may have abstained from the survey due to a variety of reasons, e.g., lack of time, unfamiliarity with or low acceptance of web-based surveys or too little experience with the decision support tools. We mitigated this limitation through the following approaches: (1) after three reminders, sending a last email to survey non-respondents asking for free-text feedback or, alternatively, a reason for why they chose to abstain, and (2) including feed-back from clinical coordinators on the acceptance of the tools among the involved healthcare providers.

We used risk matrices to assess potential risks in the pre-implementation phase. While risk matrices have limitations due to their categorical nature and potential oversimplification of risks, they can serve as a valuable tool for getting an overview of potential challenges in a large-scale implementation project when used appropriately [29, 30].

Finally, the statistics on report retrievals via the Safety Code card do not allow us to distinguish between user groups and reasons for scanning the code, e.g., patients vs. healthcare providers, testing the QR code vs. retrieving the report to use it for personalizing drug prescriptions.

Meta-evaluation & lessons learned

Previous work has pointed out that much PGx implementation work is done in isolation without sharing experiences and best practices [41]. We would therefore like to share insights, experiences and tools collected within this large-scale, multinational implementation project that can be of value to similar implementation projects.

The design and implementation of the U-PGx CDS infrastructure could not have happened without the collaboration of many contributors with diverse backgrounds and expertise, ranging from basic research to clinical PGx to software engineering. Such a multidisciplinary, large-scale and international implementation project, however, also provides challenges in terms of project management and communication.

We used a variety of communication channels throughout the implementation, including regular structured teleconference calls, structured questionnaires and dedicated mailing lists that ensured that emails were received by all relevant persons. One mailing list was established to provide a contact point for clinical sites to report issues or ask questions surrounding the genotyping platform and the CDS tools, while another one was used to enable discussions with, and distribution of information to all relevant persons at clinical sites.

Each teleconference call regarding the technical implementation was transcribed in detail and shared together with a short summary and action items with all collaborators. In the initial phase, we also experimented with using different Gantt chart software tools, such as GanttPRO [42]. While we think that the Gantt chart helped in keeping a broad overview of the different tasks and their interdependencies, we used it less for tracking and updating the more granular tasks, as this would have required every collaborator to be familiar with and actively use the tool.

We chose the online survey platform Alchemer both for conducting an initial risk assessment analysis and for the evaluation of the CDS tools among participating healthcare providers due to its piping and logic features, which helped to keep survey completion time low, and provided the possibility to track response status [31].

Conducting a collaborative and structured assessment of risks was helpful to gain an overview of potential challenges, anticipate issues and circumvent blockers that cannot easily be solved due to e.g. institutional barriers.

Opportunities and challenges regarding maintenance and extended roll-out in Europe

Within the U-PGx project, we have demonstrated the feasibility and acceptance of a framework for PGx CDS implementation that forms the basis for realizing the vision of harmonized, barrier-free PGx decision support across Europe. Still, maintenance and extended roll-out in Europe will require continued effort and work. This concerns the maintenance and improvement of the CDS framework components including the guidelines, the harmonization of PGx information in drug labels and guidelines, and, ideally, the establishment of standards including genotype-phenotype translations to guarantee interoperability and to secure quality control.

PGx guidelines

The DPWG guidelines authored and maintained by one project partner (Royal Dutch Pharmacists Association, KNMP) were used as a source for the PGx recommendations. While these guidelines have been pre-existing and had already been routinely used in primary care and hospital clinic by the Dutch implementation site before the project, resources for maintenance updating and translating guidelines were scaled up as part of the U-PGx project. Since the guidelines form the basis of PGx CDS, the securing of financial and personnel resources are a prerequisite for their further maintenance and potential international expansion.

Standardisation and harmonisation

Another essential aspect regarding interoperability and transparency concerns the standardization of PGx variants and phenotypes and the harmonization of information on PGx variants and their clinical relevance. Recent analyses have shown that substantial discrepancies exist between drug labels and guidelines in terms of content and coverage of PGx variants, and also between guidelines authored by different PGx working groups, such as the DPWG and CPIC [43-48].

Continual efforts on standardization and harmonization have been undertaken by guideline-authoring consortia, such as defining a list of consensus terms together with a panel of PGx experts and standardizing genotype to phenotype translation [49, 50]. Furthermore, a standardized minimal panel of PGx variants for pre-emptive PGx testing has recently been proposed by van der Wouden et al. [17].

Consistency of information in drug labels and guidelines is not only important for standardization but may also impact financial coverage by health insurances, and ultimately liability and authorization issues. This has recently been exemplified by the United States Food and Drug Administration’s (FDA) warning against making predictive statements about dosing for gene-drug pairs not on its authorized list which was issued to several providers of PGx tests and CDS solutions [51].

Finally, consistent and actionable PGx information in drug labels might also positively impact physicians’ willingness and motivation to consider PGx results in daily practice, as has been pointed out previously [44].

Software maintenance and compliance with MDR

Delivering large-scale PGx decision support for a large number of variants and drugs in a clinical setting requires resources for maintenance, quality control, support and regulatory compliance of software products. In the U-PGx project, these aspects were secured by the bio.logis Genetic Information Management GmbH and bio.logis digital health GmbH who already had experience in delivering PGx CDS solutions including experienced staff and the prerequisites for certification as a medical device under European Medical Device Regulation (MDR).

In Europe, while PGx CDS falls under Regulation (EU) 2017/745 [52], it is, however, not directly regulated and may therefore, depending on the concrete specifications of the system at hand and the type of algorithm used, remain in a legal grey area. While the simple matching of phenotypes with recommendations realized with a lookup table might not constitute a medical device under current legislation, the more complex matching of SNP variants and combinations with phenotypes is more likely to require CE certification when used outside of pure research settings. However, achieving CE mark certification is a time, cost and resource-consuming procedure consisting of several phases, such as obtaining a declaration of conformity and a clinical evaluation report. This may be challenging for research and clinical institutions without dedicated software development departments.

The U-PGx project partner bio.logis digital health GmbH will continue to offer their IT services beyond project end. In addition, it is planned to offer relevant software code in an open source framework. This should enable an interested community to develop further structures, work on improvements and lead to broader distribution. The quality assurance approved elements of the system are in the process of certification as a medical device class 2a. Further, as part of the maintenance strategy, an open source, light-weight version of the Safety Code card system that performs simple matching between phenotypes and recommendations and can be used as a standalone system or integrated with existing infrastructures was developed by the Medical University of Vienna [53, 54].

Future roll-out scenarios

Several future roll-out scenarios that may complement each other could be envisioned. One scenario is the piecewise roll-out of local, commercial solutions as it is already ongoing in many European countries. Advantages of this approach include the coverage of compliance with MDR, support and software maintenance and development is covered by the vendor. Potential problems include system fragmentation and lack of wide-spread trust among healthcare professionals and patients if no common standard is agreed upon or the standard is not promoted by a recognized institution.

In another scenario, a pan-European organization could take over the role of further maintaining, promoting and distributing the U-PGx decision support tools. This could be a newly created organization, an already existing European-level organization or a large and established company. While this scenario has the clear advantage of the availability of a central institution that promotes and maintains a standardized solution, it is unclear which organization or company could play such a role and how such an institution could be established.

Finally, in a third scenario, a consortium or organization that acts as a standards body could be established, publishing standards and basic solutions and working with local and national organizations. While this scenario, similarly to scenario II, has the advantage of the availability of a central institution that promotes and maintains a set of standards, it entails the challenges of building up including the ongoing acquisition of funding. The establishment of an international institution to promote global harmonization of PGx information in drug labels and guidelines has also been suggested by Koutsilieri et al. [43].

The proposed scenarios are not necessarily mutually exclusive but could complement each other, e.g., a scenario based on commercial vendors might profit from the establishment of an organization promoting standards for PGx testing and decision support to guarantee quality, transparency and standardization.

Conclusions

We have demonstrated the feasibility of implementing a standardized PGx decision support solution in a multinational, multi-language and multi-center setting. Overall, evaluation results indicated a satisfactory adoption and acceptance of the provided CDS tool among participating healthcare providers. Further, our evaluation results and process analyses will inform the improvement of the CDS framework to further lower the barrier of clinical PGx implementation in Europe and other regions of the world. Maintenance and extended roll-out across Europe will require ongoing efforts to tackle remaining challenges such as harmonization of existing PGx information in guidelines and drug labels, devising roll-out strategies that lower the barrier for healthcare institutions and providers while ensuring compliance with regulatory and legal frameworks.

Requirements analysis.

Requirements analysis framework and results.

(PDF)

Click here for additional data file.

Regulatory and legal framework analysis.

Analysis of the U-PGx CDS infrastructure in relation to the European regulatory and legal landscape.

(PDF)

Click here for additional data file.

Risk assessment.

Questionnaire to identify potential risks and problems of the implementation and results.

(PDF)

Click here for additional data file.

CDS evaluation questionnaire.

Questionnaire for evaluating the CDS tools among participating healthcare providers.

(PDF)

Click here for additional data file.

STARi checklist.

Filled Standards for Reporting Implementation Studies (StaRI) checklist.

(PDF)

Click here for additional data file.

9 Nov 2021

Pharmacogenomics decision support in the U-PGx project: Results and advice from clinical implementation across seven European countries

PLOS ONE

Dear Dr. Samwald,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Thank you for your patience during the peer review period, it was longer than expected, since it was challenging to invite appropriate peer reviewers for your manuscript. Besides considering the reviewer comments at your revision, I kindly ask you to include the name of the clinical trial registry and registration number in the Abstract.

Please submit your revised manuscript by Dec 23 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

János G. Pitter, MD, PhD

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. During our internal checks, the in-house editorial staff noted that you conducted research or obtained samples in another country. Please check the relevant national regulations and laws applying to foreign researchers and state whether you obtained the required permits and approvals. Please address this in your ethics statement in both the manuscript and submission information.

3. Thank you for stating the following in the Competing Interests section:

“DS has developed concepts of genetic information management (GIM) that are realized by the company bio.logis digital health GmbH. For the company, she is CEO. She is also medical director of diagnosticum Center of Human Genetics, a diagnostic institution that is applying and testing the GIM-systems in the context of medical care. GPP is Full Member and National representative of the European Medicines Agency, Committee for Human Medicinal Products (CHMP) - Pharmacogenomics Working Party, Amsterdam, the Netherlands. The other authors have no competing interests to declare.”

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: ""This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

4. One of the noted authors is a group or consortium Ubiquitous Pharmacogenomics Consortium. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.

5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please delete it from any other section.

Additional Editor Comments:

Please include the name of the clinical trial registry and registration number in the Abstract.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: From a randomized crossover study design, the authors report results of clinical decision support systems (CDS) implementation in the large-scale European research project Ubiquitous Pharmacogenomics (U-PGx), in which PGx CDS was rolled out and evaluated across more than 15 clinical sites. Personalized pharmacogenomics (PGx) reports were generated from 6884 genotyped samples. Participating healthcare providers expressed overall satisfactory adoption and acceptance of the CDS tools.

Minor revisions:

1- In addition to stating counts, provide corresponding percentages on the following pages: 11, 15-16, 17.

2- Line 343 indicates that averages were reported in table 3 but the column is labelled medians. Clarify.

3- Cite the statistical software used for the analysis.

4- Figure 5: Include the percentage of respondents in each country.

5- Figures 6-11: Include percentages which correspond to the counts.

Reviewer #2: Clinical Decision Support is a potentially crucial tool for widespread adoption of pharmacogenomics in clinical practice. In this manuscript, the authors analyze the rollout of a PGx CDS implementation project that spans multiple sites and countries in Europe. Strengths of the study include its multi-site, international nature and its mixed-methods approach for data collection. This is a potentially valuable study and the data collection, analysis, and results are generally well done. However, there are a number of issues with the paper's overall organization and flow, which should be corrected. Additionally, the Discussion section does not address some of the more interesting results that I expect readers could learn the most from. Feedback follows, organized by section:

Abstract

• The methods section references analysis of "workflow bottlenecks," but this terminology isn't used in the methods and results section, so it's not clear to me what this is specifically referring to.

• The majority of the Results section of the abstract reflects commentary from the Discussion section, not the results of the systematic analysis. I suggest reorganizing to make this clear to the reader and instead emphasizing the reported results.

Background

• Context should be expanded. A number of other PGx CDS programs have reported outcomes of their projects, so what are the specific literature gaps/unknowns this study addresses? U-PGx's international, multi-site nature is unique, so several can be inferred, but the authors should more explicitly state what they set out to learn with this study that's different from other PGx CDS implementation retrospectives.

Methods

• The Methods section contains editorial-type statements that are better suited to the Background or Discussion sections, including:

o A statement starting on line 129 providing a definition for "pre-emptive PGx testing"

o A statement starting on line 143 providing commentary on the necessity of PGx CDS

o A statement starting on line 217 regarding the value of risk matrices

• The Evaluation framework section lists four methods for data collection in this study. The following sub-sections explicitly define methods 2-4, but the first method, "tracking of general process indicators" is not explicitly defined. It's not clear to me what this means, what is being measured, or how the data was collected. (Additionally, the "Statistics on report retrievals" heading appears to be incorrectly formatted as a higher-level heading, but that could be an artifact of my reader – I suggest double checking the formatting to be sure.)

Results

• The "General implementation process indicators" section begins with several statements on methodology that are more appropriate for a Methods section. Reorganizing this would also help address my comment above.

• The Demographics section under "Decision support infrastructure evaluation survey" reports the demographics of those who responded to the survey, but it's not clear how representative these respondents are of overall U-PGx users. Are any statistics available on basic user demographics to put these numbers into context? For example, does a 30.8% proportion accurately reflect the number of Pharmacist users, or are they over/under-represented?

Discussion

• The statement starting on line 596 mis-characterizes the FDA's stance on PGx testing. The FDA has not banned PGx testing or CDS for gene-drug pairs that are not on their list. Rather, they have warned sellers of PGx tests and PGx CDS against making predictive statements about drug dosing that have not been verified by the FDA. Labs are still able to test PGx-related genes and CDS tools for a variety of drug-gene pairs are still available, but their predictive statements are limited. The following URL should be referenced, in addition to the tables already referenced: https://www.fda.gov/medical-devices/precision-medicine/table-pharmacogenetic-associations

• There were several interesting positive findings in the Results section that were not directly addressed in the Discussion, which I would like to see further comment on, including:

o Users reported a high level of satisfaction with the PGx training they received, which is in contrast with typical reports showing low familiarity with PGx among providers. What is it about U-PGx that led to this level of satisfaction?

o Similarly, users reported high satisfaction for workflow fit, information needs being met, and system user-friendliness. Previous projects have struggled in these areas. What did U-PGx do that made this successful?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

14 Jan 2022

We thank the reviewers and editor for helping us improve our manuscript! Replies to comments are listed below.

Editor comments:

“1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf”

→ We revised the formatting of the manuscript including the file names to comply with PLOS ONE’s requirements.

“2. During our internal checks, the in-house editorial staff noted that you conducted research or obtained samples in another country. Please check the relevant national regulations and laws applying to foreign researchers and state whether you obtained the required permits and approvals. Please address this in your ethics statement in both the manuscript and submission information.”

→ The study was approved by ethics review boards in all participating countries and all necessary permits and approvals were obtained. We added this information to the ethics statements.

We have now also included the names of ethics review boards in all participating countries.

“3. Thank you for stating the following in the Competing Interests section:

“DS has developed concepts of genetic information management (GIM) that are realized by the company bio.logis digital health GmbH. For the company, she is CEO. She is also medical director of diagnosticum Center of Human Genetics, a diagnostic institution that is applying and testing the GIM-systems in the context of medical care. GPP is Full Member and National representative of the European Medicines Agency, Committee for Human Medicinal Products (CHMP) - Pharmacogenomics Working Party, Amsterdam, the Netherlands. The other authors have no competing interests to declare.”

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: ""This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf. “

→ We included an updated competing interests statement in the cover letter.

“4. One of the noted authors is a group or consortium Ubiquitous Pharmacogenomics Consortium. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.”

→ We added the requested information to the acknowledgements section.

“5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please delete it from any other section.”

→ We moved the ethics statement to the Methods section of the manuscript.

Reviewer #1:

“From a randomized crossover study design, the authors report results of clinical decision support systems (CDS) implementation in the large-scale European research project Ubiquitous Pharmacogenomics (U-PGx), in which PGx CDS was rolled out and evaluated across more than 15 clinical sites. Personalized pharmacogenomics (PGx) reports were generated from 6884 genotyped samples. Participating healthcare providers expressed overall satisfactory adoption and acceptance of the CDS tools.

Minor revisions:

1- In addition to stating counts, provide corresponding percentages on the following pages: 11, 15-16, 17.”

→ We added the percentages in the main text.

“2- Line 343 indicates that averages were reported in table 3 but the column is labelled medians. Clarify.”

→ We corrected this by replacing ‘average’ with ‘median’ in the text.

“3- Cite the statistical software used for the analysis.”

→ We added information on the software used for conducting the analyses in the ‘Methods’ section.

“4- Figure 5: Include the percentage of respondents in each country.”

→ We extended Figure 5 to show the percentage of respondents per country.

“5- Figures 6-11: Include percentages which correspond to the counts.”

→ We updated all figures to include the percentages in addition to the counts.

Reviewer #2:

“Clinical Decision Support is a potentially crucial tool for widespread adoption of pharmacogenomics in clinical practice. In this manuscript, the authors analyze the rollout of a PGx CDS implementation project that spans multiple sites and countries in Europe. Strengths of the study include its multi-site, international nature and its mixed-methods approach for data collection. This is a potentially valuable study and the data collection, analysis, and results are generally well done. However, there are a number of issues with the paper's overall organization and flow, which should be corrected. Additionally, the Discussion section does not address some of the more interesting results that I expect readers could learn the most from. Feedback follows, organized by section:

Abstract

• The methods section references analysis of "workflow bottlenecks," but this terminology isn't used in the methods and results section, so it's not clear to me what this is specifically referring to.”

→ We added an explanation for this term in the methods section on page 12.

“• The majority of the Results section of the abstract reflects commentary from the Discussion section, not the results of the systematic analysis. I suggest reorganizing to make this clear to the reader and instead emphasizing the reported results.”

→ We re-structured and extended the abstract to highlight the results of our analyses.

“Background

• Context should be expanded. A number of other PGx CDS programs have reported outcomes of their projects, so what are the specific literature gaps/unknowns this study addresses? U-PGx's international, multi-site nature is unique, so several can be inferred, but the authors should more explicitly state what they set out to learn with this study that's different from other PGx CDS implementation retrospectives.”

→ We extended the ‘Background’ section to more explicitly describe how our study differentiates from existing CDS implementation projects and what our study adds to the existing literature. Indeed we especially highlight the multi-national, multi-site, multi-lingual and multi-modal nature of our intervention and study.

“Methods

• The Methods section contains editorial-type statements that are better suited to the Background or Discussion sections, including:

o A statement starting on line 129 providing a definition for "pre-emptive PGx testing"”

→ We moved the section describing the setting of the study including the definition of preemptive PGx testing to the ‘Background’ section.

“o A statement starting on line 143 providing commentary on the necessity of PGx CDS”

→ We removed the corresponding sentence.

“o A statement starting on line 217 regarding the value of risk matrices”

→ We moved the statement on risk matrices to the ‘Discussion’ section.

“• The Evaluation framework section lists four methods for data collection in this study. The following sub-sections explicitly define methods 2-4, but the first method, "tracking of general process indicators" is not explicitly defined. It's not clear to me what this means, what is being measured, or how the data was collected.”

→ We added a paragraph to the ‘Methods’ section that describes the process indicators and how these data were collected.

“(Additionally, the "Statistics on report retrievals" heading appears to be incorrectly formatted as a higher-level heading, but that could be an artifact of my reader – I suggest double checking the formatting to be sure.)”

→ Indeed it was incorrectly formatted; we corrected the formatting of the heading.

“Results

• The "General implementation process indicators" section begins with several statements on methodology that are more appropriate for a Methods section. Reorganizing this would also help address my comment above.”

→ We moved these explanatory statements to the newly added paragraph on ‘process indicators’ in the ‘Methods’ section (see our reply above).

“• The Demographics section under "Decision support infrastructure evaluation survey" reports the demographics of those who responded to the survey, but it's not clear how representative these respondents are of overall U-PGx users. Are any statistics available on basic user demographics to put these numbers into context? For example, does a 30.8% proportion accurately reflect the number of Pharmacist users, or are they over/under-represented?”

→ We added information on demographic characteristics of the overall invited healthcare providers that were available to us in the ‘Demographics’ section to make it clear how representative our sample is for the total number of users. The gender ratio of respondents matches that of invited participants. Based on the institutions where invited participants were employed, we infer that pharmacists were somewhat over-represented and primary care physicians were somewhat under-represented among respondents. However, these fluctuations do not significantly impact our overall findings, since all groups were sufficiently covered by the pool of respondents.

“Discussion

• The statement starting on line 596 mis-characterizes the FDA's stance on PGx testing. The FDA has not banned PGx testing or CDS for gene-drug pairs that are not on their list. Rather, they have warned sellers of PGx tests and PGx CDS against making predictive statements about drug dosing that have not been verified by the FDA. Labs are still able to test PGx-related genes and CDS tools for a variety of drug-gene pairs are still available, but their predictive statements are limited. The following URL should be referenced, in addition to the tables already referenced: https://www.fda.gov/medical-devices/precision-medicine/table-pharmacogenetic-associations”

→ Thanks for pointing this out! We adapted the text to more accurately reflect the FDA’s stance. We noticed that the link referenced in this comment is the same as the one already cited in the references.

“• There were several interesting positive findings in the Results section that were not directly addressed in the Discussion, which I would like to see further comment on, including:

o Users reported a high level of satisfaction with the PGx training they received, which is in contrast with typical reports showing low familiarity with PGx among providers. What is it about U-PGx that led to this level of satisfaction?”

→ We added a paragraph in the discussion section that provides possible explanations for the high level of satisfaction with PGx CDS training.

“o Similarly, users reported high satisfaction for workflow fit, information needs being met, and system user-friendliness. Previous projects have struggled in these areas. What did U-PGx do that made this successful?”

→ We extended the discussion section to provide potential explanations for these positive findings compared to previous PGx CDS implementation projects.

Submitted filename: Reply to reviewers.pdf

Click here for additional data file.

3 May 2022

Pharmacogenomics decision support in the U-PGx project: Results and advice from clinical implementation across seven European countries

PONE-D-21-24987R1

Dear Dr. Samwald,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

János G. Pitter, MD, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Thank you for your patience in the peer review period, it took longer than expected to secure the necessary peer reviews.

Reviewers' comments:

26 May 2022

PONE-D-21-24987R1

Pharmacogenomics decision support in the U-PGx project: Results and advice from clinical implementation across seven European countries

Dear Dr. Samwald:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. János G. Pitter

Academic Editor

PLOS ONE