Improving the Safety of Medicines via Digital Technology: An Assessment of the Scope and Quality of Risk Minimization Websites in the United States and United Kingdom.

INTRODUCTION: EHealth holds tremendous promise for enhancing drug safety initiatives known as risk minimization programs. Little is known, however, regarding the scope and quality of existing risk minimization websites.
METHODS: Two publicly accessible repositories, REMS@FDA [1] and Electronic Medicines Compendium [2], were reviewed to identify all regulatorily approved risk minimization programs in the United States (US) and United Kingdom (UK) with websites. Website quality was evaluated using the Enlight Quality Assessment tool, a psychometrically validated instrument that addresses seven quality domains.
RESULTS: Ninety-three websites were identified: 59 for healthcare professionals (7 UK/52 US), and 34 for patients (5 UK/29 US). The websites functioned chiefly as archives for electronic copies of educational materials; a subset (31/93) had additional features. Mean quality ratings for Usability (mean 4.70, SD 0.59), Visual Design (mean 4.03, SD 0.87) and Content (mean 4.31, SD 0.82) were good. General Subjective Evaluation was fair (mean 3.15, SD 1.21). Mean scores for Therapeutic Alliance and Therapeutic Persuasiveness were poor (mean 2.62, SD 1.47; and mean 2.50, SD 1.48, respectively); those for User Engagement were very poor (mean 2.25, SD 1.03). No differences were found by target audience but several were identified based on region.
CONCLUSIONS: Risk minimization websites are easy to navigate and well organized. Few, however, incorporate eHealth design elements that facilitate user engagement, build therapeutic alliance and exert therapeutic persuasiveness. Such elements can enhance program uptake and effectiveness. Results highlight opportunities for improving the quality of risk minimization websites and their ability to bridge pharmaceutical and healthcare systems.

Key Points

Introduction

Digital technologies are revolutionizing many aspects of healthcare, from the conduct of medical research to the delivery of clinical services [3, 4]. Nowhere is this transformation more evident than in the sphere of drug development. Digital tools are now being used to expedite patient recruitment into clinical trials, to elicit informed consent, to collect data for novel study endpoints, to monitor trial participants’ status remotely and to aid in clinical diagnosis and decision support [3]. Digital technologies are also being used in the post-marketing context to support patient adherence to drug regimens, to characterize real-world use of medicines by patients and associated health outcomes, and to enhance the patient’s experience using a medicinal product [5].

Digital technologies are now being applied for drug safety purposes as well. For example, efforts are underway to digitize drug product labels to enhance their accuracy, timeliness, and accessibility to healthcare professionals and patients [6] [7]. Beyond the product label, which is considered to be a ‘routine risk minimization measure,’ other drug safety initiatives, known as ‘additional risk minimization measures’ (aRMMs), can benefit from harnessing digital technologies as well [7-9].

In most instances, routine risk minimization measures are considered adequate to ensure a positive benefit–risk profile for a medicine. However, in situations where routine measures are not deemed to be sufficient, aRMMs (also known as ‘risk minimization programs’) may be required in order, for example, to manage a particularly serious risk [7]. These risk minimization programs represent a type of public health intervention designed to “prevent or reduce the occurrence of adverse reactions associated with exposure to a medicine, or to reduce their severity or impact on the patient should adverse reactions occur” [8]. These programs typically consist of one or more tools and activities including (i) educational materials (e.g., healthcare professional [HCP] or patient guides, drug self-administration demonstration kits, risk awareness forms, Patient Cards, HCP checklists, or other types of clinical decision aides), (ii) specific communication measures targeted at HCPs (e.g., Dear Healthcare Professional Letters); and (iii) controlled drug distribution measures involving, for example, healthcare and/or patient training and certification [8, 9].

Marketing Authorization Holders (MAHs) are required to design risk minimization programs, facilitate their implementation into the healthcare delivery system, and evaluate their effectiveness. Evaluations should assess the following key domains: (i) process of implementation (i.e., the extent to which the program activities and materials were delivered to the target audience(s) in the right amount and at the right time), (ii) proximal outcomes (i.e., changes in knowledge, attitudes, and behaviors), (iii) distal outcomes (i.e., number and type of serious adverse drug reactions; health status; other clinical indicators), (iv) program burden on the healthcare system, and (v) program maintenance or sustainability [8, 10]. Evaluations of risk minimization programs to date have revealed shortcomings in program design and implementation as well as mixed or uncertain effectiveness [11-16]. As a result, improving both the quality and effectiveness of these programs has emerged as a high priority for regulators [17, 18].

In the broader field of public health, a digital health intervention is defined as one which “uses information and communications technology in support of health” [19]. The term ‘digital’ encompasses both electronic health (eHealth) and mobile health (mHealth) technologies. In particular, eHealth interventions typically consist of a dedicated website or electronic platform. The website may perform a range of functions and host an array of features, including informational resources and activities. The exact website functionalities and features may vary depending on the intervention goals and can range from the very basic (e.g., information provision via posting of electronic versions of educational materials) to the highly sophisticated (e.g., simulations of real-world use environments for skills training) [20, 21]. In this paper, we use the terms ‘eHealth program’ or ‘eHealth intervention’ to refer to a website containing a set of functionalities and features intended to achieve one or more specific public health goals (e.g., reduce medication errors associated with the use of a prescription medicine; ensure that patients receive appropriate vaccinations before receiving a particular medicinal product).

The past decade has witnessed a proliferation in the number of eHealth interventions aimed at addressing a wide range of health-related conditions [22]. This development reflects the growing recognition of eHealth’s potential for improving the accessibility and impact of health interventions [23]. Quality criteria for eHealth interventions have been developed to assist public health researchers and practitioners in selecting among this growing array of options [24]. These criteria address a range of domains that have been shown empirically to enhance intervention uptake, use, and therapeutic impact [24, 25].

Criteria-based rating scales represent one way to assess the quality of an intervention and to gauge whether it is fit for purpose. Such scales feature a set of criteria associated with core quality domains or constructs. In the context of medicinal product drug safety interventions, the application of criteria-based assessment scales represents a cost-effective way for both sponsors and regulators to evaluate the quality of proposed risk minimization programs. One example of this can be seen in the use of a criteria-based rating scale to evaluate the quality of written risk minimization materials for products with approved Risk Evaluation and Mitigation Strategies (REMS) [26].

While the potential value of digital technologies for improving both the reach and effectiveness of risk minimization programs has been acknowledged by regulators [8, 27], no explicit regulatory guidance has been issued to date regarding how to either design or evaluate eHealth risk minimization programs [8, 10]. Additionally, neither the extent to which eHealth risk minimization programs are being used currently, nor the quality and potential effectiveness of the programs that do exist, has been documented. To address these gaps, this study sought to (i) identify the number of approved eHealth risk minimization programs; (ii) describe the types of features offered as part of those programs; and (iii) assess each program’s quality.

Methods

Identification of eHealth Risk Minimization Programs

Data Sources

To identify digital or ‘eHealth’ risk minimization programs (referred to as ‘risk minimization websites’ hereafter), we reviewed two repositories where approved risk minimization programs are posted: the Electronic Medicines Compendium (EMC) [2] in the United Kingdom (UK) and the REMS@FDA platform [1] in the United States (US). With the exception of the Ireland Health Products Regulatory Authority’s platform, these are the only extant, English language-based repositories of this type that are publicly accessible. The Electronic Medicines Compendium (EMC) is managed independently by Datapharm as a subscription service. While companies are not mandated to subscribe to EMC, a significant number (i.e., 250+) of pharmaceutical companies are currently registered as such [2]. EMC contains information about medicines licensed for use in the UK and approved by either the Medicines and Healthcare Products Regulatory Agency (MHRA) or the European Medicines Agency (EMA) (EMC http://www.medicines.org.uk/emc). The repository is organized into categories by active substance within which there are separate entries for each product by manufacturer, including both innovator and biosimilar/generic products. It is the responsibility of prescribers to ensure that their entries are accurate and kept updated. Up until January 2021, the UK was part of the European Economic Area (EEA) and hence under the jurisdiction of the EMA. As a result, the majority of all products approved in the EEA would have been incorporated into the EMC, making the repository representative of approved risk minimization programs across the EEA region up until that date.

The REMS@FDA is a website that is maintained by the US Food and Drug Administration (FDA), where all approved REMS for products marketed in the US, and their associated risk minimization tools, are posted. In contrast to the EMC site, the REMS@FDA site contains information concerning approved and active REMS only.

Review Method

An initial review was conducted to (i) determine whether each drug product had an approved risk minimization program (relevant for the EMC database only), and (ii) to ascertain whether the approved risk minimization program was available in a digital format. All risk minimization websites that had active uniform resource locators (URLs) and were accessible by all members of the research team were considered eligible for study inclusion. In the second step of the analysis, the quality of each of the digital risk minimization websites was evaluated. Four raters (JF, SF, RM, MYS) participated in the evaluation activity, all of whom were subject matter experts in therapeutic risk minimization. For assessment purposes, each website was rated independently by two different raters (a rater ‘dyad’). The four raters were grouped to form six unique rater dyads with each dyad assigned 14–18 websites to review. Following completion of all website assessments, a calibration exercise was conducted. This exercise involved comparing the scores on each item between raters. If differences between raters were >2 points on any one item, individual raters explained their rationale for assigning a given rating and raters were given the opportunity to re-rate the item within a 2-point range.

Enlight Quality Assessment Scale

We used the Quality Assessment section of the Enlight instrument to evaluate the quality and therapeutic potential of the risk minimization websites identified from our initial review. Enlight is a suite of checklists and rating scales that can be used to classify and appraise eHealth interventions in terms of target audience and clinical goals, credibility, privacy and security features, and quality.

The Enlight Quality Assessment scale was developed based on a systematic review of quality criteria for eHealth and mHealth interventions developed in the context of medical products where users were expected to utilize the interventions for medical reasons [24]. It was designed to assess intervention domains pertaining to user experience quality and therapeutic potential. The scale features 25 items across seven domains (subscales) (Table 1). Items are rated using a 5-point ordinal scale (1 = very poor; 2 = poor; 3 = fair; 4 = good; 5 = very good). The total assessment score for each subscale is calculated by summing the scores for all items within each subscale and averaging the sum total. The Enlight Quality Assessment scale has been psychometrically evaluated and shown to have both strong construct validity and inter-rater reliability [24].

Table 1

The Enlight eHealth Quality Assessment Scale: subscale domains and domain descriptions

Subscale domain	Description of concepts within domain
1. Usability	a. Navigation b. Learnability c. Ease of use
2. Visual design	a. Aesthetics b. Lay-out c. Size
3. User engagement	a. Content presentation b. Interactive c. Not irritating d. Targeted/tailored/personalized e. Captivating
4. Content	a. Evidence-based content b. Quality of information provision c. Complete and concise d. Clarity about program’s purpose
5. Behavior change/persuasive design	a. Call for action b. Load reduction of activities c. Rewards d. Real data driven/adaptive e. Ongoing feedback f. Expectations and relevance g. Therapeutic rationale and pathway
6. Therapeutic alliance	a. Basic acceptance and support b. Positive therapeutic expectations c. Relatability
7. General evaluation (appropriateness of features)	a. Appropriate features to meet clinical aim b. Right mix of ability and motivation c. Likeability

Analysis of Results of Risk Minimization Websites’ Quality Assessments

Analysis of the results of the risk minimization websites’ quality assessments followed a multi-step process. First, scale ranges (lowest to highest possible score) were examined. Next, the means and standard deviations of each individual Enlight subscale and subscale item was calculated, both overall for all the eligible programs as well as by region (UK versus US websites) and by target audience (HCPs versus patients). Mean subscale and subscale item scores were interpreted as follows: very poor (mean score 1.00–1.99); poor (mean score 2.00–2.99); fair (mean score 3.00–3.99); good (mean score 4.00–4.99); and very good (mean of 5.00).

Lastly, to assess inter-rater reliability, we calculated the percentage agreement between the two raters within each dyad for each subscale and the corresponding weighted kappas. The weighted kappa was deemed the appropriate statistic due to the ordinal nature of the response scales [28]. The weighted kappa values were interpreted according to Cohen’s heuristic [29].

Results

Review of Approved Risk Minimization eHealth Programs

Electronic Medicines Compendium (EMC)

The EMC review was conducted in February, 2021. All product entries were reviewed for each active substance listed. Three hundred and twenty-two of the medicines in the repository were identified as having at least one form of aRMM. Datapharm confirmed that, as of 22 April 2021, the number of medicines listed in the repository was 9760 [email on file]. Thus, 322/9760 (3.3%) of the medicines had at least one form of aRMM recorded. The vast majority of medicines with aRMMs (305/322 [94.7%]) had only downloadable portable document format (PDF) versions of their risk minimization materials posted individually on the EMC portal itself.

A small number of medicines (17/322 [5.3%]) had an eHealth risk minimization program in the form of a dedicated website that included one or more features (Table 2). Eight of the 17 programs and associated websites were for eight different active substances. Of the nine remaining programs, five were for generic methylphenidate products that all shared the same risk minimization program and associated website, and four were for generic abacavir products that similarly shared the same program and associated website.

Table 2

The Electronic Medicines Compendium and the REMS@FDA databases: eHealth risk minimization websites as a proportion of all approved risk minimization programs and types of website features

	Number of product entries	Percentage
EMC
Total number of products listed in EMC	9760
Total number of products with an approved risk minimization program	322	3.3
Number of products with an eHealth risk minimization program	17	5.3
Number of eHealth programs	10a
Features of the eHealth risk minimization programsb
^Expiry alert service	3	30
^Feedback survey	4	40
^Instruction/administration video	4	40
^Knowledge test	3	30
^Postings of PDF copies of paper-based educational materials only	2	20
REMS
Total number of approved REMS programs	60
Number of REMS that include an eHealth risk minimization program	58	96.7
Features of the eHealth risk minimization programsb
^Mobile applicationc	3	5
^Feedback survey	0	0
^Knowledge test	13	22
^Instruction/administration video	3	5
^Postings of PDF copies of paper-based educational materials only	39	67

aTwo eHealth risk minimization programs, representing 5 and 4 medicines respectively, shared the same application programming interface (API)

beHealth risk minimization programs may be counted multiple times if they featured more than one digital feature. The total number of eHealth risk minimization programs that contained one or more additional programmatic features beyond posting PDF copies of educational materials only was 12

cOne app covering 3 APIs—a platform for app delivery of additional risk minimization measures

Of the ten unique eHealth programs identified for the 17 medicines, two featured a home page where PDF copies of the aRMMs were available only. Eight others included one or more additional features. Examples of the latter included instructional or administration videos and an automated expiry alert service.

REMS@FDA

At the time of review (February 2021) there were 60 active REMS on the REMS@FDA site. In contrast to the EMC, the REMS@FDA repository grouped all REMS associated with a given active substance together (including both the reference product and generic products) under one listing as part of a ‘shared system’ REMS.

Of the 60 active REMS, the vast majority (58/60) included a dedicated risk minimization eHealth website. Of these REMS websites, the majority (39/58 [67.2%]) consisted of a home page where only PDF versions of paper-based educational materials were posted. Approximately one-third (19/58) of the websites featured one or more additional features, including three REMs (from the same pharmaceutical company) which included an app designed to provide immediate access to all of the risk minimization materials for the given manufacturers’ products.

Assessment of the Quality of eHealth Risk Minimization Programs

Figure 1 presents a flow diagram showing how the eHealth risk minimization programs were selected for analysis purposes. The dataset was structured such that shared eHealth programs for generic/biosimilars in both the US and UK were only counted once. A total of 111 eHealth risk minimization programs were identified for initial review. Of these, 71 were for HCPs (14 UK/57 US) and 40 were for patients (5 UK/35 US). Five HCP programs and six patient programs were subsequently excluded due to the fact that either the website link was no longer active or it was not accessible to all members of the research team. Seven additional HCP eHealth programs (UK-based) were excluded due to the fact that they were duplicate sites for generic products. The final set of eligible eHealth programs included 59 for HCPs (7 UK/52 US) and 34 for patients (5 UK/29 US).

Fig. 1

Flow diagram of risk minimization eHealth website review. HCP healthcare professional

Each rater rated a minimum of 43 websites; several slightly exceeded this minimum (i.e., Rater 3 reviewed 47 websites; Raters 2 and 4 rated 48 websites each). Quality ratings performed by Rater 1 were on average the lowest (mean 83.30), while quality ratings performed by Rater 2 were the highest (mean 90.56) (Supplementary Table 1, see the electronic supplementary material [ESM]). Examination of intra-dyad results for each item revealed that out of the total 5022 dyad comparisons, 1.97% had intra-dyad differences > 2 points. Following reconciliation, scores for these affected items were recalibrated.

Table 3 presents the mean and standard deviation for all seven quality subscales for the eHealth programs overall as well as by region (UK versus US) and target audience (patients versus HCPs). For the eHealth programs overall, mean quality ratings were good for the Usability subscale (mean 4.70, SD 0.59), Visual Design (mean 4.03, SD 0.87) and Content (mean 4.31, SD 0.82) subscales. The mean rating for the General Subjective Evaluation subscale was fair (mean 3.15, SD 1.21). In contrast, mean scores for the remaining three subscales ranged from poor to very poor. Specifically, mean ratings for the Therapeutic Alliance and Therapeutic Persuasiveness subscales fell into the poor range (mean 2.62, SD 1.47; and mean 2.50, SD 1.48, respectively). The mean ratings for the User Engagement subscale were the lowest and ranked as very poor (mean 2.25, SD 1.03).

Table 3

Mean and standard deviations (SD) for each of the Enlight subscales and subscale items for all risk minimization eHealth websites, and by region and target audience

Enlighta Subscale and subscale items	EHealth websites by region				EHealth websites by target audience				All eHealth websites (n = 93)
	UK (N = 12)		US (N = 81)		Patient (N = 34)		HCP (N = 59)
	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Usability	4.85	0.43	4.68	0.37	4.76	0.56	4.66	0.59	4.70	0.59
Ease of navigation	4.92	0.28	4.72	0.62	4.81	0.58	4.71	0.60	4.75	0.59
Learnability	4.83	0.47	4.72	0.49	4.82	0.38	4.69	0.53	4.74	0.49
Ease of use	4.79	0.50	4.59	0.67	4.66	0.68	4.58	0.64	4.61	0.66
Visual design	4.04	0.95	4.03	0.91	4.04	0.92	4.02	0.84	4.03	0.87
Aesthetic design	3.79	1.04	3.83	0.84	3.78	0.95	3.85	0.82	3.82	0.87
Layout	4.21	0.87	4.07	0.79	4.12	0.83	4.07	0.78	4.09	0.80
Size	4.13	0.88	4.19	0.90	4.22	0.92	4.15	0.89	4.18	0.90
User engagement	2.80	1.28	2.17	0.96	2.27	1.07	2.24	1.01	2.25	1.03
Content presentation	3.63	1.11	2.93	0.80	3.03	0.86	3.02	0.89	3.02	0.88
Interactive	2.42	1.19	1.37	0.66	1.54	0.90	1.48	0.78	1.51	0.82
Targeted/tailored/personalized	2.33	1.21	2.06	0.84	2.10	0.97	2.09	0.86	2.10	0.90
Captivating	2.83	1.18	2.31	0.83	2.40	0.97	2.37	0.85	2.38	0.90
Content	4.35	0.78	4.30	0.82	4.26	0.88	4.34	0.78	4.31	0.82
Evidence-based content	4.83	0.37	4.82	0.38	4.82	0.38	4.82	0.38	4.82	0.38
Information provision quality	4.38	0.81	3.86	0.79	3.84	0.92	3.97	0.74	3.92	0.81
Complete and concise	4.33	0.55	4.28	0.85	4.25	0.85	4.31	0.80	4.29	0.82
Clarity about program’s purpose	3.88	1.01	4.24	0.86	4.12	0.93	4.24	0.86	4.19	0.89
Therapeutic persuasiveness	2.72	1.61	2.46	1.46	2.48	1.50	2.51	1.47	2.50	1.48
Call for action	3.21	1.08	2.60	0.80	2.69	0.90	2.68	0.85	2.68	0.87
Load reduction of therapeutic goals	4.58	0.57	4.25	0.99	4.15	1.18	4.37	0.79	4.29	0.96
Therapeutic rationale and pathway	3.75	1.13	3.67	1.00	3.59	1.17	3.74	0.92	3.68	1.02
Reward for meeting goals	1.21	0.71	1.07	0.30	1.16	0.53	1.04	0.24	1.09	0.38
Data driven/adaptive	1.04	0.20	1.13	0.39	1.06	0.24	1.15	0.42	1.12	0.37
Ongoing feedback	1.54	1.15	1.19	0.47	1.31	0.83	1.19	0.45	1.23	0.62
Expectations and relevance	3.71	1.14	3.35	0.98	3.43	1.08	3.37	0.97	3.39	1.01
Therapeutic alliance	3.14	1.52	2.54	1.46	2.60	1.51	2.63	1.46	2.62	1.47
Basic acceptance and support	2.79	1.32	1.98	0.74	2.06	1.01	2.10	0.80	2.09	0.88
Confidence, positive therapeutic expectations	4.21	0.82	4.23	0.93	4.15	1.03	4.28	0.83	4.23	0.91
Relatability	2.42	1.58	1.40	0.69	1.59	1.00	1.50	0.87	1.53	0.92
General subjective evaluation	3.65	0.93	3.07	1.23	3.09	1.25	3.18	1.18	3.15	1.21
Appropriate features to meet the clinical aim	4.29	0.68	4.36	0.78	4.28	0.87	4.40	0.70	4.35	0.77
Right mix of ability versus motivation	3.29	0.84	2.23	0.84	2.31	0.94	2.41	0.89	2.37	0.91
I like the program	3.38	0.81	2.62	0.82	2.68	0.93	2.75	0.80	2.72	0.85

HCP Healthcare professional, SD standard deviation, UK United Kingdom, US United States

No appreciable quality differences were seen by target audience type. However, several differences were seen in the quality of the eHealth programs by region. Specifically, the quality of information provision was rated as being higher for UK eHealth programs (mean 4.38, SD 0.81) than for those in the US (mean 3.84, SD 0.79). In contrast, clarity of purpose was rated higher for US programs (mean 4.24, SD 0.86) as compared with those in the UK (mean 3.88, SD 1.01). Therapeutic persuasiveness was rated higher for UK versus US programs both in terms of basic acceptance and support (UK mean 2.79, SD 1.32; US mean 1.98, SD 0.74) as well as in relatedness (UK mean 2.42, SD 1.58; US mean 1.40, SD 0.59).

In the Therapeutic Alliance domain, call for action was rated higher for the UK than for the US websites (mean 3.21, SD1.08; mean 2.60, SD 0.80, respectively). In the domain of General Subjective Evaluation, the UK programs were rated as being of higher quality than the US in terms of features to meet clinical aim (UK mean 3.29, SD 0.84; US mean 2.23, SD 0.84), and right mix of ability and motivation (UK mean 3.38, SD 0.81; US mean 2.62, SD, 0.82). Lastly, while User Engagement ratings were on average very poor overall, the UK websites were rated higher than those in the US in terms of interactivity (UK mean 2.42, SD 1.19; US mean 1.37, SD 0.66) and content presentation (UK mean 3.63, SD 1.11; US mean 2.93, SD 0.80).

Table 4 presents the percent agreement and weighted kappa statistics across raters for all of the websites. Agreement among the four raters was moderate (weighted kappas: 0.51–0.57) for rater dyads 1–3 [29]. Agreement was good to substantial for rater dyads 4–6 (weighted kappas: 0.67–0.75).

Table 4

Percent agreement and weighted kappa statistics for 27 items in the Enlighta Quality Assessment tool for each rater dyadb

	Dyad 1 (Rater 1/Rater 2)	Dyad 2 (Rater 1/Rater 3)	Dyad 3 (Rater 1/Rater 4)	Dyad 4 (Rater 2/Rater 3)	Dyad 5 (Rater 3/Rater 4)	Dyad 6 (Rater 2/Rater 4)	Total no. of comparisons completed
No. of websites	12	16	15	18	14	18	NA
No. of comparisonsc	324	432	405	486	378	486	2511
Weighted kappa	0.55	0.57	0.51	0.67	0.68	0.75	NA
95% CI	0.49–0.60	0.51–0.61	0.46–0.56	0.60–0.74	0.61–0.74	0.67–0.84	NA
% agreement	41.7%	40.7%	40.5%	56.3%	53.1%	60.5%	NA

NA not applicable CI confidence interval

aBaumel A, Faber K, Mathur N, Kane JM, Muench F. Enlight: A Comprehensive Quality and Therapeutic Potential Evaluation Tool for Mobile and Web-Based eHealth Interventions. J Med Internet Res. 2017. 19(3): p. e82

bRater 1: JF; Rater 2: MYS; Rater 3: SF; Rater 4: RM

cThe number of comparisons between raters was calculated as the number of websites rated multiplied by 27 (i.e., the number of items to rate in each website)

Figure 2 shows the percentage frequency of quality ratings for each item overall across the seven scale domains. Consistent with the Enlight Quality Scale scoring, a quality rating of 1 was termed as being ‘very poor,’ a rating of 2, ‘poor,’ a rating of 3, ‘fair’, a rating of 4 as ‘good’ and a rating of 5 as ‘very good.’ Results show that in terms of the Usability domain (including the items navigability [1a], learnability [1b], and ease of use [1c]), websites were consistently rated as being very good (i.e., >70%). Similarly, in terms of the Content domain, websites were also consistently highly rated, especially the degree to which content was deemed to be evidence-based (4a), complete and concise (4c), and clear (4d) as to the program’s purpose. In terms of the domains Therapeutic Persuasiveness and Therapeutic Alliance, two items specifically ‘load reduction of activities’ (5b) (i.e., Are the therapeutic activities sufficiently simple? Do the features make it as easy as possible to complete the activities?), and ‘positive treatment expectations’ (6b) (i.e., Does the eHealth program encourage users to expect beneficial outcomes from utilizing the program and to rely upon it in a medical context?) were rated high in most instances (i.e., the combined good and very good scores were 68% and 78% for items 5b and 6b, respectively).

Fig. 2

Distribution of quality ratings per item on the Enlight Quality Assessment Scale for all risk minimization eHealth websites (n = 93)

In contrast, items in the User Engagement domain, especially the degree of interactivity (item 3b), targeting and tailoring of content (item 3c), and degree to which the eHealth program was deemed as providing a captivating user experience (item 3d), were consistently rated as being poor or very poor in most instances. In the domain Therapeutic Persuasiveness, the items ‘call to action’ (5a), ‘rewards’ (item 5d), ‘data driven and adaptive content’ (item 5e), and ‘ongoing feedback’ (item 5f) were consistently rated as being of poor to very poor quality. Similarly, in the domain Therapeutic Alliance, such items as ‘basic acceptance and support’ (item 6a) and ‘relatability’ (item 6c) (i.e., a good representation of a human factor, such as a professional or peer, that is easily relatable within the therapeutic context process) were rated as being poor to very poor in terms of quality in most instances.

In terms of the final domain Overall Subjective Evaluation, the item ‘appropriate features to meet the clinical aim’ (item 7a) (i.e., Are the eHealth program’s features sufficient enough to meet its therapeutic goals?) was consistently highly rated in terms of quality. However, the items ‘right mix of ability and motivation’ (item 7b) and ‘I like the program’ (item 7c) received low quality ratings in most instances.

A similar pattern of results was seen when the eHealth program quality ratings were evaluated separately by target audience (HCPs versus patients) (Figs 3 and 4, respectively).

Fig. 3

Quality ratings of eHealth risk minimization websites for healthcare professionals (n = 59): United States and United Kingdom

Fig. 4

Quality ratings of patient eHealth risk minimization websites (n = 34): United States and United Kingdom

Discussion

To the best of our knowledge, this is the first quality assessment of risk minimization websites for medicinal products conducted to date. Overall, we found that these websites were functioning primarily as repositories for electronic PDF copies of paper-based educational materials—thus serving, in essence, as digital ‘pamphlet racks’ [20]. Only a small subset of the websites (approximately one-third) had any additional features. Such features, however, were limited and fell into three categories: (i) instructional videos (e.g., how to self-inject asthma medication), (ii) the ability to set product expiry notifications (e.g., for asthma inhalers), and (iii) user-administered knowledge checks regarding drug-related risks and safe use procedures. These findings indicate that the vast majority of the risk minimization websites relied exclusively on a simple information-transfer model to achieve their programmatic goals [14]. While websites of this type are quicker, easier and less costly to design, they are also less effective from an intervention standpoint [14].

In our quality assessment, we focused on a set of seven eHealth quality domains that have been empirically found to be associated with intervention adherence and therapeutic effectiveness [24]. Overall, the risk minimization websites received good ratings in terms of Usability and Visual Design; they were generally easy to use and navigate, and acceptable in terms of visual aesthetics, layout, and size of fonts and graphics. Content, specifically the degree to which the content of the information on the eHealth website was evidence-based, clear, and concise, was also rated as good. Despite this, however, both the quality of information provision and clarity regarding the program’s purpose were deemed lacking, thus undercutting their ability to make a compelling ‘call to action’ to end users.

In terms of the three remaining quality domains (i.e., Therapeutic Persuasiveness, Therapeutic Alliance, and User Engagement), the websites were rated as being poor to very poor. Persuasive design features are derived from a psychological understanding of what influences individual behavior and decision making in regard to health. The use of such features has been shown to promote higher user engagement with eHealth interventions, and beneficial health-related behavior change [24, 30, 31]. Four main categories of persuasive design features have been identified: (i) primary task support (helping the user do the main task such as through the ability to repeat or rehearse it); (ii) system credibility support (perceived trustworthiness of the tool and/or tool content); (iii) dialogue support (reminders, rewards, reinforcing messages); and (iv) social support (e.g., via competition or social comparison, or recognition) [32, 33].

Therapeutic Alliance refers to features that nurture positive therapeutic expectations [24]. Therapeutic alliance has been shown to increase participant engagement and effectiveness of eHealth interventions [34]. One way that the therapeutic alliance can be strengthened is by making the risk minimization program more relatable to the user. Examples of ways to strengthen therapeutic alliance include posting an educational video by an actual healthcare professional (as opposed to a virtual representation), or sharing patient testimonials. Therapeutic alliance can also be enhanced by establishing a sense of partnership between the participant and the intervention itself such as by providing participants with encouraging, accepting and supportive feedback [24, 34].

User engagement or interactivity was the least used, and hence most under-developed feature, in all of the risk minimization websites. This finding has implications for program effectiveness. Specifically, four types of interactive design features have been shown to be associated with greater program impact: (i) social context and support (e.g., synchronous and asynchronous-mediated peer-to-peer communication), (ii) contacts regarding the intervention (e.g., expert-initiated contact delivering behavior change techniques or prompts such as motivational email messages or reminders), (iii) self-management (e.g., individual goal setting or action planning), and (iv) interactive tailoring so that the intervention is more responsive to the unique needs, motivations and personal characteristics of the user [23]. In regard to interactive tailoring, research has shown that individuals differ markedly in terms of how they seek and access health information, and in the depth of information that they desire [35]. Outcomes of such information-seeking efforts are best when individuals are able to obtain the type and amount of information they prefer. Thus, website designs that offer users different options for accessing information can better accommodate the full spectrum of information-seeking preferences, ranging from the minimum (for those concerned about information overload) to the in-depth (for those desiring fuller details) [32]. Collectively, the ultimate goal of tailoring and interactive design features is to achieve effective user engagement—engagement that is of sufficient intensity and duration to increase the likelihood of achieving therapeutic impact [36].

We found no notable differences in quality ratings across the identified risk minimization websites based on target audience type. However, we did find differences based on region. The proportion of total approved risk minimization programs (in either the UK or US) with a dedicated website was markedly higher in the US than in the UK. This may be due to differences in regulatory jurisdiction between the two regions. Specifically, many of the UK risk minimization programs were for products that had been centrally approved by the EMA and hence were also intended for use throughout the EEA. Under EU regulations, responsibility for oversight of risk minimization programs is shared between the EMA and local National Competent Authorities (NCAs). Specifically, the EMA is charged with approving a description of the risk minimization measures in the risk management plan, including which tools will be used and the key messages to be conveyed. In contrast, NCAs are charged with approving the actual content of these measures, their format, and the distribution channel(s) used to implement them at the local country level. The approval of the risk minimization implementation plan falls to the local NCA as well [7, 8]. Additionally, applicable regulations concerning sponsor websites are complex and vary from country to country in the EEA. Given that risk minimization websites, however limited in functionality, entail considerably greater development time, upfront financial investment, and regulatory scrutiny than their traditional, non-digital counterparts, it is understandable why sponsors might opt for the latter approach.

In addition to differences in the proportion of risk minimization websites between the UK and the US, differences in quality were seen as well. Specifically, the UK risk minimization websites were of superior quality in terms of therapeutic alliance, therapeutic persuasiveness, overall subjective evaluation and information provision. In comparison, the US websites were rated higher on one item only: clarity concerning the website’s purpose. The latter finding may reflect the fact that the FDA has provided standard templates for designing REMS programs, templates that include an initial section where a statement is required concerning the program’s goals and objectives [10].

In both regulatory jurisdictions, existing regulations concerning promotional materials have affected how risk minimization measures are designed. As a further complication, there is no agreement as to the definition of ‘promotional’ amongst regulators at present. Guidance from both the FDA and EMA emphasize that risk minimization measures should neither resemble, nor be used as, promotional materials. In practice, this has been interpreted to mean that risk minimization measures should be minimalistic in terms of design aesthetics, including limited use of color, pictures, or illustrations and other visual elements. Such an approach, however, is not empirically supported and, in fact, runs counter to the goal of risk minimization. Visually attractive materials are more likely to engage the attention of patients and healthcare professionals, and to increase the likelihood that they read and refer back to the risk minimization information as needed.

A related challenge is that EU legislation forbids pharmaceutical companies from communicating to patients directly, in contrast to the practice in the US. As a result, the responsibility for successfully conveying the risk minimization materials and messages to patients falls almost exclusively to healthcare professionals. Dissemination of these materials would be greatly enhanced if multiple trusted channels were used instead of just one.

Currently, efforts are underway to address these barriers. Patient groups are articulating the need for change as reflected, for example, in the forthcoming CIOMS XI report on patient involvement in drug development and safe use of medicines. Additionally, the EU pharmaceutical legislation is now open for updating, and improving medicinal product risk communication and risk minimization are among the prioritized topics to be addressed.

Risk minimization programs are complex interventions: they target multiple audiences (i.e., patients, prescribing physicians, other healthcare professionals), involve many different elements, must be implemented across a range of different healthcare settings, geographic locations and regulatory jurisdictions, and address a diverse set of outcomes (e.g., drug safety knowledge, behaviors, and clinical endpoints) [37]. EHealth technologies offer enormous opportunities for improving the uptake, delivery and impact of risk minimization programs. Moreover, adoption of these technologies is only likely to accelerate as a result of the green agenda and efforts to improve sustainability and reduce environmental waste. As our study results indicate, however, the potential value of such technologies for risk minimization purposes has yet to be fully exploited, especially in regard to behavioral change outcomes.

Several examples of digital risk minimization tools that include more sophisticated eHealth features do exist, although none (to the best of our knowledge) have been incorporated into formally approved risk minimization programs. These tools include (i) a web-based platform, connected with a widely used continuing medical education website, that offers a risk minimization educational program and an accompanying interactive knowledge test [38]; (ii) automatic counseling reminders embedded within electronic medical records that prompt physicians to perform risk counseling and to provide patients with certain risk minimization educational materials [39]; and (iii) a web-based, interactive training tool for pharmacists and pharmacist technicians that enables them to practice specific admixing techniques so as to reduce the potential for error in preparing an infusion-delivered oncology product [39, 40].

We propose several recommendations to improve the quality of eHealth approaches to risk minimization and to advance the science in this area. These recommendations are especially intended for those pharmacovigilance professionals who are novices in the design and application of eHealth applications for risk minimization.

First, program planners should use theory-informed design frameworks to guide the design of eHealth risk minimization interventions to improve the likelihood of their uptake and, ultimately, their effectiveness in clinical care [23, 41, 42]. The value of using a framework is that it offers a systematic, transparent, and comprehensive approach to designing a risk minimization intervention (digital or otherwise), one that is linked to theory and to a body of empirical research. Many frameworks exist and several have been identified as being especially relevant for not only designing, but for implementing and evaluating risk minimization programs [42].

Second, program planners should incorporate human-centered design approaches into the development of risk minimization websites and other associated digital risk minimization tools. Human-centered design seeks to leverage a deep understanding of the end user’s knowledge, skills, behavior, motivations, and context to inform the design of the intervention. By definition, it entails the involvement of end users (e.g., patients, healthcare providers, informal caregivers) throughout the development and testing process. Human-centered design approaches have been shown to improve the feasibility, acceptance and adoption of eHealth interventions in real-world contexts [40, 43].

Third, in order to increase uptake of risk minimization websites, planners should consider employing multi-faceted implementation strategies to promote awareness and uptake of these interventions in different healthcare settings and among different types of users [42]. An extensive taxonomy of implementation strategies is available [44].

Fourth, program developers should build in multiple, periodic assessments of the eHealth intervention, including both formative and summative evaluations. Formative evaluations are those which are conducted during the development process. Their goal typically is to determine the acceptability of the eHealth intervention to the end user (e.g., usefulness, visual appeal), the feasibility of use in real-world contexts, and the extent to which the end user can easily access the information, guidance or training that s/he is seeking. Typically, formative evaluations are iterative and dynamic. They can involve a variety of approaches, including usability testing, contextual inquiry, interviews, and ethnographic observations. Evaluation should continue during the implementation phase and again at several points post-launch (i.e., summative evaluations) to assess the intervention’s initial and sustained impact over time [45]. Consistent with GVP guidance [8], planners should assess a range of process and outcome indicators, including those that can be derived from the backend analytics captured routinely on eHealth websites. Examples of process metrics include the number of unique website ‘hits’ (visits) within a specified time period, the average amount of time spent on each section of the site, and the number of unique user downloads of risk minimization materials. Examples of outcome measures include results from online knowledge tests, and behavioral evaluations that assess the extent to which the individual can perform a certain (simulated) action or calculation (e.g., inject a syringe at the specified angle; determine the appropriate weight-based dose of a medicine).

Risk minimization program planners are encouraged to be proactive and explore the need for digital risk minimization tools and websites during the phase III period of product development so that design activities can be commenced at that point if needed. This approach may entail some initial ‘at risk’ investment; however, this risk can be balanced by the fact that, if a formal risk minimization program is deemed necessary by regulators, the program will be ready for submission, thus avoiding a delay in obtaining marketing approval.

This study had several limitations. First, our analysis was restricted to eHealth interventions only. In the course of our review, we identified only three instances in which an mHealth tool (mobile app) had been developed—too small a number to include in our quality assessment analysis. Nonetheless, as the number of mHealth applications increases, it will be important to assess their quality, particularly in light of evidence suggesting that mHealth interventions have a potential advantage over eHealth interventions in terms of their ability to reach, engage, and persuade different target audiences [24].

A second limitation was that our study results cannot be generalized to all approved eHealth risk minimization websites globally as we had access only to those that are currently extant in the US and the UK. As noted previously, however, the eHealth programs we reviewed in the UK were likely highly representative of those approved for use in Europe as a whole as the UK was part of the EEA up until January of 2021. Thirdly, there was a slight time discrepancy between ascertainment of the numerator (number of products with approved risk minimization programs) and confirmation of the true size of the denominator (total number of products listed) in the EMC repository. However, the time interval was very short (approximately 2 months) so it is not expected to have affected the accuracy of the percentage calculation by more than 1%.

An additional possible limitation is that the study dataset was prepared such that shared eHealth programs for generics, biosimilars, and a reference product were only counted once. This was especially relevant in analyzing data from the EMC which listed websites separately for each product, regardless of its status (i.e., as an innovator or biosimilar/generic). We opted for this approach because the websites of both the reference and biosimilar/generic products were required to be essentially the same, and separate ratings would have resulted in overweighting ratings for sites with active substances that had a greater number of biosimilar/generic products. The strength of our chosen analytic approach was that the final ratings were representative of the eHealth interventions developed for a particular active substance, regardless of whether that included one or several medicines.

Conclusion

Our findings show that existing risk minimization websites are being used predominantly as repositories for electronic versions of risk minimization materials. As a corollary to this, we found that while the websites earned good marks generally for usability and visual design, they received poor ratings in terms of their ability to engage users, foster therapeutic alliance, and be therapeutically persuasive.

These findings have several implications. First, by failing to exploit the full range of functionalities offered by digital technology, risk minimization websites have diminished potential to be effective as interventions. In particular, timely uptake and engagement with the website content and materials are essential if a risk minimization program is to be successful in affecting knowledge, attitudes, behaviors, and ultimately in reducing the targeted product risks.

Second, both eHealth and mHealth digital technologies are predicted to play a substantial role in shaping drug development, public health, and health care delivery in the twenty-first century. Given this, risk minimization interventions will increasingly need to use digital technologies, especially if they are to successfully integrate into the healthcare delivery system and reduce burden on healthcare professionals and patients, both of which are stated regulatory priorities.

Third, while we can expect a hybrid approach to risk minimization—paper-based approaches plus digital platforms and other tools—in the near term, risk minimization websites will likely emerge as one of the dominant risk minimization tools in the future. In addition to web-based applications, mobile (mHealth) tools will gain greater uptake as well for risk minimization purposes. Pharmaceutical companies that can develop digital tools which incorporate specific, evidence-based features to facilitate ease of use, encourage effective user engagement, and promote trust and therapeutic alliance will have a competitive edge in that their risk minimization programs will be more successful in terms of reaching healthcare professionals and patients as well as in achieving the desired risk minimization outcomes.

A final implication is that risk minimization programs, consistent with the needs of a learning health system and agile science, should be evaluated for effectiveness on an ongoing basis in a manner that is pragmatic, timely, and not resource intensive. Digital approaches to risk minimization offer a range of new indicators for assessing both process and outcome measures that are both relatively easy and inexpensive to collect. Similarly, conducting a quality criteria assessment as part of a formative evaluation, such as described here using the Enlight Quality Assessment tool, offers a relatively quick and low-cost way to assess the potential uptake and effectiveness of a risk minimization website during the design phase, thus permitting further modifications to be made prior to its actual real-world implementation.

Considering the enormous potential of eHealth interventions in supporting the safe and appropriate use of medicinal products, ongoing research in this area is vital. Specifically, further empirical investigation is needed to determine how, why, and in what contexts specific eHealth design features affect risk minimization program outcomes. Additionally, research is needed to identify which intervention features are most effective in encouraging user engagement, building therapeutic alliance, promoting trust, and creating the intended behavioral change. Not least, program evaluators should seek to publish their findings in peer-reviewed journals to build the evidence base and help advance the science of drug safety.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 18 KB)

Supplementary file2 (DOCX 36 KB)

Digital websites are important drug safety tools as they offer patients and healthcare professionals ready access to key risk minimization resources for medicinal products.

This study was the first of its kind to assess the quality of existing risk minimization websites in the United States and the United Kingdom.

Results showed that while the websites were typically well organized and easy to search, they had limited functionality and served primarily as repositories for electronic educational materials.

Moving forward, risk minimization program developers should seek to take fuller advantage of digital technology by incorporating website design features that promote and sustain engagement and motivate users to adopt safe use behaviors.