Recommendations on the Use of Mobile Applications for the Collection and Communication of Pharmaceutical Product Safety Information: Lessons from IMI WEB-RADR.

Over a period of 3 years, the European Union's Innovative Medicines Initiative WEB-RADR (Recognising Adverse Drug Reactions; https://web-radr.eu/ ) project explored the value of two digital tools for pharmacovigilance (PV): mobile applications (apps) for reporting the adverse effects of drugs and social media data for its contribution to safety signalling. The ultimate intent of WEB-RADR was to provide policy, technical and ethical recommendations on how to develop and implement such digital tools to enhance patient safety. Recommendations relating to the use of mobile apps for PV are summarised in this paper. There is a presumption amongst at least some patients and healthcare professionals that information ought to be accessed and reported from any setting, including mobile apps. WEB-RADR has focused on the use of such technology for reporting suspected adverse drug reactions and for broadcasting safety information to its users, i.e. two-way risk communication. Three apps were developed and publicly launched within Europe as part of the WEB-RADR project and subsequently assessed by a range of stakeholders to determine their value as effective tools for improving patient safety; a fourth generic app was later piloted in two African countries. The recommendations from the development and evaluation of the European apps are presented here with supporting considerations, rationales and caveats as well as suggested areas for further research.

Key Points

Introduction

The Innovative Medicines Initiative (IMI) WEB-RADR (Recognising Adverse Drug Reactions) consortium was a public–private partnership supported by the IMI Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115632. Participating members were from European regulatory agencies, the pharmaceutical industry, academia, patient groups and other organisations with an interest in pharmacovigilance (PV). The aim of WEB-RADR was to develop and evaluate two digital tools to support PV activities and promote patient safety, and ultimately provide policy, technical and ethical recommendations on implementing such technology. Full details of IMI WEB-RADR’s objectives and approaches have been detailed elsewhere [1]. The outputs from WEB-RADR arose from four work packages: two work packages undertook original research in social media and mobile application (app) technology; a third evaluated the scientific impact of the original research to determine where it had potential to add value to existing PV methodologies. A fourth work package addressed governance and policy, including personal data protection and ethical and societal considerations related to the use of mobile apps and social media for PV. This paper will focus on recommendations resulting from the work on apps. Recommendations resulting from the research conducted using social media will be the subject of a separate publication.

The profusion of smartphones and other mobile devices offers an opportunity to further engage the public in reporting suspected adverse drug reactions (ADRs) to national competent authorities (NCAs) and to improve communication of emerging safety issues from NCAs to the public and healthcare professionals (HCPs). Together with web-based forms, mobile devices offer a platform for developing real-time PV systems that can enable near-instantaneous transmission of patient safety information at the point of need, potentially improving health outcomes. Data collection in PV typically relies on voluntary reporting of suspected ADRs submitted to NCAs and pharmaceutical companies by HCPs. Over the last 20 years, patient reporting has been increasingly recognised as a valuable addition to safety monitoring activities [2, 3]. This growing interest in the patient perspective has coincided with a proliferation of technology, which in turn has increased access to data, accelerated information exchange, and fostered transparency. It is therefore reasonable to assume that efforts to improve ADR reporting and patient safety communication could benefit from the adoption of new technologies such as mobile apps.

Early experiences with mobile apps for medical product safety have been promising and have demonstrated potential for broad engagement of the public in safety reporting [4]. At the same time, concerns over security and potential for misuse of personal data have grown with several well-publicised breaches such as those at Yahoo, CEX and Facebook [5-7]. Aside from developing and evaluating digital tools themselves, the WEB-RADR project also examined the ethical, societal and privacy concerns related to mobile apps, as well as technological and promotional aspects, from a European perspective.

The WEB-RADR mobile apps were designed to facilitate instantaneous reporting of suspected ADRs directly to NCAs by patients and HCPs, as well as timely dissemination of accurate PV information from NCAs to HCPs and the public, thereby allowing potential two-way exchange of safety information. The apps have been made freely available by PV centres in the United Kingdom (UK; Medicines and Healthcare products Regulatory Agency [MHRA]), the Netherlands (NL; Lareb), and Croatia (HR; HALMED) in both Google Android and Apple iOS formats; links to download the apps are available at https://web-radr.eu/mobile-applications-for-adr-submission. A fourth version of the app has been piloted in Burkina Faso and Zambia in collaboration with the World Health Organization (WHO); at the time of writing, this fourth version was still undergoing validation and assessment.

The use of mobile apps for ADR reporting and the downstream use of these data for PV is a relatively new and unexplored concept. The anticipated impact of these apps will depend on the degree of uptake by the public and HCPs, the quality and volume of ADRs reported, and the utility of the data in safety signal detection and evaluation. In WEB-RADR, each NCA devised its own strategy for mobile app promotion best suited to its national context and providing a range of engagement models for assessment. Focus group discussions and face-to-face interviews were conducted with potential app users from multiple countries [8]. Enthusiasm for the app was high and the feedback received was constructive, targeted and generally encouraging. Comments from the participants have been used to inform further app development to improve uptake and encourage continued use of the app. As of 31 December 2017, there have been over 17,000 app downloads and 838 suspect ADR reports received since the first app was launched on 14 July 2015 (Table 1).

Table 1

Breakdown of app downloads and reported ADRs by country to 31 December 2017a

	United Kingdom (MHRA)	Netherlands (Lareb)	Croatia (HALMED)
Date of app launch	14 July 2015	29 Jan 2016	18 May 2016
Number of ADR reports	505	173	160
App downloads for iOS	7498	4172	422
App downloads for Android	2701	2497	876

ADR adverse drug reaction, app application, MHRA Medicines and Healthcare products Regulatory Agency

aAs of Q4 2017, mobile operating system penetration within these three countries was as follows: UK: Android 46.7%, iOS 50.6%; the Netherlands: Android 56.0%, iOS 41.9%; and Croatia: Android 82.3%, iOS 15.0%. See http://gs.statcounter.com/os-market-share/mobile

To integrate ADR reports submitted via the apps into existing ADR processing workflows at NCAs, a direct connection was made between each app and the respective NCA’s existing electronic gateway. When a user submits an ADR report through the app, the reported data are automatically converted into standard International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) E2B(R2) format, allowing it to be submitted directly to the NCA’s PV portal for triage, assessment and signal detection alongside ADR reports received through other reporting channels. The apps also provide a means for NCAs to broadcast to users personalised safety alerts and news about user-selected drugs. App users can also explore aggregate ADR report information in interactive graphs that display volumes of reports received by the NCA (at the national level in the UK and NL and at the global or EU level for HR) for medicines of interest according to Medical Dictionary for Regulatory Activities (MedDRA®; https://www.meddra.org) system organ class and, where available, by patient demographics such as age and sex. NCAs submitted suspected ADRs from the mobile apps to EudraVigilance by batching app reports with reports submitted through other channels, in accordance with usual reporting requirements [9].

The following recommendations1 relate to policy considerations and original research pertaining to the development and utility of a mobile app platform for PV conducted under the auspices of the IMI WEB-RADR project between September 2014 and September 2017. In total, we present 27 separate recommendations with the intention of informing PV professionals, particularly those with an interest in the development of digital technologies to enable PV.

Summary of the Research and Recommendations

The recommendations outlined in this paper have been developed based on evidence from the activities undertaken through IMI WEB-RADR. For a full description of the studies, refer to the cited original publication or technical report. For the sake of brevity, unless explicitly stated to the contrary, the terms ‘mobile app’ (or simply ‘app’) are used to cover software designed for mobile devices, irrespective of the means of presentation, i.e. via a smartphone or a tablet. Recommendations were drafted by the authors of the various outputs that collectively described the outcomes of the app work package. Draft recommendations were subjected to several rounds of review by other work package members, resulting in either their revision or rejection. Recommendations were typically rejected on the grounds that they were too general in nature or not adequately supported by insights arising from the work of WEB-RADR or if they were considered not implementable in practice. Final-draft recommendations were circulated to the WEB-RADR General Advisory Board2 for review and comment. Finally, all consortium participants were invited to review a mature draft of this paper.

Drivers for Mobile Application Development and Other Pre-development Considerations

Prior to committing the effort and investment needed to develop any mobile app, it is important to gain an understanding of the setting in which the app is to be used and the intended target group. Prior knowledge of technical and regulatory limitations and likely barriers to uptake and usage can inform app design and implementation. The recommendations provided in this section could be deemed good practice for app development in any setting; however, it is pertinent to share the specific experiences from WEB-RADR as a means of emphasising the value of undertaking a scoping exercise prior to full-scale app development. The value versus opportunity cost of app development has not been factored into these recommendations since it is likely to vary from country to country given differences such as availability of established means of ADR reporting, infrastructure maturity (internet, mobile networks, etc.), education and awareness of healthcare systems generally and PV more specifically.

Consideration of the intended use of the technology is also important. For example, the key benefits in the EU settings evaluated were related to easy access to trusted regulatory information and the impact in terms of reporting considered additive to the suite of tools available to reporters [8]. However, early feedback from the pilots in Burkina Faso and Zambia points to the value of off-line functionality (reporting and news) independent of access to a website.

Furthermore, there are efficiencies in the costs associated with standardised/shared platforms/code, with significantly reduced maintenance costs associated with the app version used in Burkina Faso and Zambia developed towards the end of the WEB-RADR project. Technology now enables the use of common code to deliver the same functionality or multiple modalities; this offers a more flexible and cost-effective framework to deliver on user needs independent of the end user device. See Table 2 for app development recommendations.

Table 2

Recommendations for application (app) development

Recommendation	Rationale
Prior to designing a mobile app, it is recommended that smartphone and device use among target populations is assessed as this can inform basic decisions regarding development	Decisions over which reporting modalities (e.g. app, website, etc.) and operating systems are supported should account for the prevalence of wired versus wireless connectivity and the devices in use (e.g. iOS vs Android vs PC) in the target setting. An evaluation should be conducted to ensure that the platform proposed is the most effective means of delivering the functionality to the target audienceFor example, smartphone penetration and use of different types of devices vary greatly from country to country. Android devices are more commonly used in Europe, South America, and Asia, while iOS devices are more commonly used in Australia and in North America, use of both operating systems is broadly even [10]. Additionally, while two-thirds of the global population use the Internet or own a smartphone, this is less common among adults in Africa and South Asia [11]
Organisations deploying apps should be aware of the differing regulations in different regions and countries	The General Data Protection Regulation (GDPR) introduces EU-wide legislation on personal data and security and developers of apps need to be familiar with the rules and obligations. In particular, they should assess the impact of data protection at the time of design concept and review compliance periodically. In addition, an ePrivacy Regulation has been proposed [12]. It is also important that developers pay attention to any specific national requirements, and testing of application security is advisedOther regulations should be considered. Although the apps developed by WEB-RADR are not currently impacted by EU Medical Device regulation, it would be advisable to maintain awareness of device regulations, particularly if the apps were to be enhanced to an extent where medical device regulations may become relevant. In case of doubt, legal advice should be sought [13]
Any new app or subsequent updates to the app or operating platform should be extensively tested before release to ensure that technical issues are avoided and to assure the app design remains intuitive, does not inhibit usage and minimises user error	A survey indicated that users experienced difficulties downloading some versions of the app. New software requires testing and validation to ensure that it will perform as expected on release. A formal validation model requiring user specification and technical specification tested using a formal validation strategy with a final report should be employed
Consultation with target user groups should be considered when undertaking app development efforts for pharmacovigilance	Prior to app launch, WEB-RADR had access to patients who contributed important suggestions based on their experience testing the app prototype. Focus groups and face-to-face interviews were conducted in multiple countries where feedback was documented formally to later inform app design [8]. Consultation may be less important if targeting well characterised populations of users or similar geographic regions

Mobile Application Design

Traditional reporting mechanisms require significant time and effort to complete and submit, inhibiting ADR report submission to Marketing Authorisation Holders (MAHs) and regulators [14]. Mobile app design for PV offers the opportunity to improve upon traditional ADR-reporting forms for patients and HCPs alike. The app design process in WEB-RADR focused on developing an uncluttered visual design incorporating feedback from patient and HCP groups, while maintaining enough consistency to enable comparison with suspected ADRs obtained through other mechanisms. Since the reports submitted through the app would be passed along to NCAs, the app screens had to include the minimum necessary data fields to comply with ICH E2B(R2) standards and requirements in line with the guideline on good pharmacovigilance practices (GVP) Module VI, which served as one of the starting points for their design [15]. The four mandatory reporting elements were the focus of data collection: (1) patient information, (2) reporter information, (3) suspected medicinal product and (4) suspected adverse reaction.

The digital platform afforded opportunities to have adaptive forms. Since ADR reports contain information about medical products and ADRs, structured report fields may require knowledge of medical terminology; however, it is unlikely that all patients will be familiar with this type of vocabulary. In the UK, work was done as part of WEB-RADR to investigate the use of patient-friendly MedDRA® terms for reporting ADRs, and a list of patient-friendly terms is available on the MedDRA Maintenance and Support Services Organization (MSSO) website [16].

There are also several features that could be integrated into an app to facilitate processing and analysis by NCAs, including structured and hierarchical event ontologies. Autofill features, an adaptive suggestion that appears in a field automatically when a user types a few letters, for symptoms and product names may expedite the report submission process and ensure that these data points are spelled correctly and adhere to standardised terminologies.

While structured text fields such as drop-down menus or radio buttons promote data standardisation, these may not be ideal in every situation. For example, patients may not comprehend ontology-based medical terms suggested in a drop-down menu or they may find that the options do not best describe their experiences. From a cognitive design perspective, overly structured fields may negate useful nuance in the patient’s experience, reducing the usefulness of the report. Early testing suggested that patients may find that completing free-text fields to describe ADRs in their own words preferable to selecting options from drop-down menus with multiple choice and radio buttons [17]. Such design considerations can conceivably impact report completeness and accuracy.

It is important to ensure that a mobile PV app can be used by as wide a population as possible. Technology industry standards can provide guidance on design elements such as preferred font size for legibility, preferred colours or shades for text and image visibility and button placement for touchscreens. Adhering to such standards as much as possible ensures that a mobile app can be used by patients and HCPs with a variety of abilities. Additionally, when designing an app for ADR reporting, any constraints imposed by regulatory requirements, database or technology limitations and resource implications should be carefully evaluated.

Beyond the visual design of the app, the digital environment permits the creation of a positive feedback loop whereby reporters receive confirmation and positive reinforcement for submitting their report via email or in-app messaging. In early experiences with PV reporting, such feedback mechanisms were believed to have improved the user experience [4]. This also creates a conduit for follow-up reporting to NCAs as well as a mechanism for push notifications when new safety alerts are issued by NCAs. Some elements of this approach to enhancing user engagement were assessed in WEB-RADR.

Since the base WEB-RADR mobile app was to be deployed in multiple countries, yet evaluated centrally by the project, a certain level of design flexibility was required to allow country-specific language and branding. In some countries the NCA’s ADR-reporting programme already has a recognisable identity and the app was therefore designed to be extensible with colours, logos, etc. to mimic extant reporting forms and branding. See Table 3 for design recommendations.

Table 3

Recommendations for application (app) design

Recommendation	Rationale
General design recommendations applicable to health apps
If a mobile app is to have different target groups (e.g. HCPs and members of the public) customisation for each target group should be considered	WEB-RADR explored the use of different report designs tailored for different types of users. For example, the users of the UK’s Yellow Card app can select MedDRA terms from a drop-down list to describe their ADR, ensuring that the reaction data are already coded when they are submitted to the MHRA. For the patient version, the MHRA is currently exploring ways to make this feature easier for patients to use with a patient-friendly MedDRA list that contains descriptions of medical concepts in layman’s terms [17, 18]. Additionally, distinct questions could be presented to different types of users who may be more likely to be able to provide certain details; for example, a patient may be able to report that a medical product was purchased from an Internet pharmacy, whereas that patient’s HCP may not have that level of knowledge
If different versions of a mobile app are needed to best meet the differing needs of multiple target groups, it is recommended that a single app prototype is developed with as many standard features as possible. This can then be configured at lower overall cost	In WEB-RADR, three apps were developed for three pilot member states, all sharing a similar framework. Ultimately, for the second and third pilots in the Netherlands (Lareb) and Croatia (HALMED), the NCAs worked together to determine a set of common design standards to create a more generic app that would require less development than the first pilot in the UK (MHRA). The Lareb app was developed and launched 6 months after Yellow Card; the HALMED app was developed and launched 3 months after LarebThe lower cost of using established applications has contributed to roll-out of the app to lower income countries; see the WEB-RADR portal (https://webradr.files.wordpress.com/2017/09/web-radr-stakeholder-event_theme2.pdf)
In order to provide a user-friendly experience for as many users as possible, ensure that the app design complies with accessibility requirements or industry best practices that facilitate use of the app among patients and HCPs with different abilities	Best practices exist for ensuring maximum accessibility through web and app design. Both Apple and Android devices have documented design principles for developers [19, 20]. In addition, the European Commission has published a set of Web Content Accessibility Guidelines from the Web Accessibility Initiative [21]. Furthermore, many modern devices and operating systems have built-in accessibility features. It will be necessary for app developers to understand how the app can take advantage of and interact with these features, rather than ignore or override them
The mobile app should give users the option of receiving push notifications from the health authority or PV centre when new safety information is available	In a quantitative survey study, more than half of respondents indicated interest in an optional push notification feature [22]. However, not all users expressed interest in this feature, for example, patients in Croatia were generally less interested, and there were also national differences in preference amongst HCPs
Mobile apps that provide safety information should include the ability for the user to configure or customise the type of information that s/he receives (e.g. information at the individual product level or the therapeutic area level)	The quantitative survey study found that users were interested in receiving safety news regarding a wide variety of medicines. HCPs generally preferred to receive information on all approved drugs, while the largest group (40%) of patients preferred to receive only information pertaining to their own prescriptions [22]
Mobile apps for ADR communication should provide users with the option of bypassing the login screen to automatically access the app after the information has been entered once	The quantitative survey study and feedback from the Netherlands app indicated that most patients and professionals preferred to access the app without having to login each time they opened it (57% and 70%, respectively) [22]. In addition, respondents to a user experience survey indicated that they would prefer not to create an account or enter login details. Providing patients and HCPs with the option to choose whether to login automatically would allow users to access the app quickly and easily—removing a hurdle to real-time ADR reporting and facilitating quick access to timely medicine news. However, should app developers choose to take this approach, they should work closely with stakeholders to determine the appropriate level of patient consent or privacy protection required for app functions that will not require password protection
Mobile apps for ADR communication should be designed with the possibility to integrate with other health/drug safety applications and/or systems through, e.g. APIs, since this will extend their uptake by a wider range of organisations and a larger group of users, consequently enhancing sustainability	Several organisations including NHS trusts, EHR system providers and developers of health apps have approached MHRA with an interest in integration of different components of the WEB-RADR app into their own platforms. Their rationale is that such an approach enables them to provide validated forms or data in a format agreed by the regulatory authority, but without the need for them to develop, maintain and update the code themselves
Specific design recommendations applicable to ADR-reporting apps
At a minimum, mobile reporting apps should provide an immediate acknowledgement message indicating that an ADR report has been successfully received	Many responders to both a quantitative survey study and a qualitative study expressed interest in receiving feedback or a confirmation following ADR report submission. Users appreciated that, upon report submission, the WEB-RADR apps displayed a simple notification informing the user that the report had been successfully sent. Earlier research by Lareb indicated a simple acknowledgement of receipt would satisfy most reporters [23]Some users indicated that additional types of feedback would be appreciated (e.g. an overview of how frequently the ADR had been reported previously; feedback on what was done with the report), while others preferred to not receive this type of response as they found it intrusive. Therefore, giving users the option to select their preferences regarding ADR feedback within the app ought to be considered [8]. Also see experiences from the SCOPE project for feedback on report templates from reporters [24]
Mobile apps for ADR reporting should be designed with an emphasis on two-way information exchange features, as this may encourage adoption and increase the likelihood of continuous user engagement and repeated use of the app	Findings from a qualitative study revealed that patients would view the news feature in the app favourably. Patients indicated that they might use the app as a quick and accessible source of information about various ADRs. For example, 84% of patients and 71% of HCPs specified that they would want to receive safety notifications on medicines from their local NCA [22]
When considering which functionalities to add to an app targeting HCPs, it is suggested to focus on features that facilitate the provision of safety information rather than features facilitating ADR report submission	According to the quantitative study, HCPs expressed a greater interest in app features that would provide them with safety information than features that would enable ADR reporting [22]
Strive for a report form design that captures all essential information in a user-friendly manner. For instance, free-text fields may be more appropriate for some questions than drop-down menus with a long list of options. It may be possible to reduce the time it takes to complete a report by keeping mandatory fields to a minimum. User-friendly phrases and descriptions may help to ensure that ADR reports are as accurate and complete as possible	Lareb operates a web-based ADR report form that includes many structured fields (e.g. radio buttons, drop-down menus). While this ensures data completeness, Lareb has also received complaints from users that reporting is time consuming. In addition, patients might find that their experiences are not well reflected in drop-down menu options and may find it preferable to provide this custom information themselvesIn order to keep the length of the ADR-reporting form to a minimum, it was shown that a shortened form in the app still produces reports with sufficient clinical quality. The proportion of reports of at least moderate quality was high in both user groups (HCPs and patients) for all countries, but relatively lower for app reports: 83% vs 92% in the UK (p = 0.08); 85% vs 98% in the Netherlands (p < 0.01); and 78% vs 78% in Croatia (p = 1.0) [25]Although free-text fields may provide patients and HCPs with the opportunity to describe ADRs in their own words, unstructured text resulting from these fields will need to be automatically coded or may require additional manual review [25]
For mobile apps aimed at HCPs, it is recommended to keep adverse reaction report forms concise and clinically focused, in contrast to those provided for patients	The qualitative study found that time constraints were more likely to be a barrier to ADR reporting among HCPs than among patients [8]
The ability to add attachments to reports is considered valuable—with the implementation of the ICH E2B(R3) format, this will become feasible and should be considered for future apps or upgrades	Mobile devices typically have built-in cameras, potentially making it easy for app users to include a photo or video along with ADR reports. Photos or videos could help capture product information (brand name, batch number, barcode) or provide a visual reference for certain ADRs (e.g. a photo of a skin rash, a video recording movement related to dyskinesia). For several years, Lareb has allowed reporters to attach additional files to ADRs submitted via web-forms. This not only decreases the burden on the reporter by allowing a simple way to relay medical information (e.g. uploading a medication list or a discharge letter from the hospital), but also increases Lareb’s understanding of the experienced ADR. The ICH E2B(R3) format that supports the transmission of additional files was not available for use during the WEB-RADR project. It follows that changes to databases and appropriate processes for receipt and analysis of such data would be needed to exploit information captured through these device features

ADR adverse drug reaction, API application programming interface, EHR Electronic Health Record, HCP healthcare professional, ICH International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use, MedDRA Medical Dictionary for Regulatory Activities, MHRA Medicines and Healthcare products Regulatory Agency, NCA national competent authority, NHS National Health Service, PV pharmacovigilance, SCOPE Strengthening Collaboration for Operating Pharmacovigilance in Europe

Mobile Application Content and Data-Related Considerations

The WEB-RADR app has been developed to improve the two-way exchange of information (i.e. the reporting of ADRs and the provision of patient safety information). Therefore, the app contains an information-sharing component and an ADR-reporting component. The information-sharing component enables app users to receive safety alerts and news, as well as view aggregated statistics on previously reported ADRs.

A qualitative study that involved focus group discussions and face-to-face interviews showed that patients and HCPs were generally positive about the existing content of the app and suggested some additional functions that could also be useful [8]. Perspectives on the content of the app were further assessed in a large online survey among patients and HCPs in Europe [22]. It was shown that potential app users were somewhat more interested in an app for two-way exchange of safety information than in an app that only served one purpose (i.e. either just the information-sharing component or just the ADR-reporting component). The survey provided many suggestions on how to further improve the app to increase ADR reporting to NCAs.

A second online survey was conducted among current users of the WEB-RADR app. The survey asked about their experiences with the app. Most responders had a positive view of the value of the app in general, for both the ADR-reporting feature and the patient safety information provided in the app. Some suggestions were also received on how to further improve the app.

A quantitative survey study found that respondents from different countries preferred access to different types of information, e.g. drug stock levels (Croatia), alternative medicines (the Netherlands), and ADR mitigation strategies (the UK). Information on newly identified drug–drug interactions was of common interest across the three countries surveyed [8, 22]. This echoes our earlier recommendation in the pre-development considerations section (Sect. 2.1): when determining what kind of information to provide to users in the app, consideration should be given to user- and country-specific content. Interests among different user groups could differ, and countries may vary greatly in terms of what safety information may need to be broadcast; for example, local regulatory communication of newly identified drug–drug interactions may vary where there are formulary differences across countries. See Table 4 for a summary of the recommendations regarding content and data-related considerations.

Table 4

Recommendations regarding application (app) content and data-related considerations

Recommendation	Rationale
The ability to access all information contained in Patient Information Leaflets (or Product/Patient Package Inserts) should be considered as an additional feature of any app designed for patient use	The quantitative survey study found that patients who were interested in other app functions expressed most interest in being able to use the app to access both Patient Information Leaflets and an overview of previously reported ADRs, when presented with both these possibilities [22]. However, this approach should only be considered where the information can be appropriately formatted and indexed for use on a mobile device
The app should provide users with the ability to access, save a draft and review their own submitted reports as well as summary information regarding ADRs previously reported to the NCA	Users preferred having the ability to look at their previous ADR reports. In addition, patients indicated that a major benefit of the app would be if it allowed them to check whether a symptom has previously been reported as an ADR. Several studies have shown that patients are sometimes uncertain about an association between a symptom and a drug and having this information in the app could reduce their uncertainty about an association between a drug and their symptoms [8, 22, 26–29]. To respect users’ desire for anonymity, however, there should exist an option not to store personal details when saving a report
Apps should provide information to users about the importance of ADR reporting. For patients, this should contain an explanation of its importance for patient safety and that the information they voluntarily submit can benefit other stakeholders	The quantitative study found that patients were mainly motivated to report ADRs to contribute to patient safety knowledge and to share their experiences for the benefit of others [22]

ADR adverse drug reaction, NCA national competent authority

Mobile Application Implementation, Awareness and Uptake

NCAs can use various strategies to inform potential users about the existence of the mobile app, but it is most efficient to use those channels and sources that reach the audience with potentially the highest interest in the app.

To assess which patients and HCPs are most interested in an app with capabilities for the two-way exchange of safety information, data from a European-wide online survey study were used [22]. In the survey, patients and HCPs were asked to what extent they were interested in the app. Analyses of the HCP data showed that HCPs who already (at least sometimes) use a health app were very interested in the app. This was also shown among the patients. In addition, patients that were younger were very interested in the app.

See Table 5 for a summary of recommendations on implementation, awareness and uptake.

Table 5

Recommendations on application (app) implementation, awareness and uptake

Recommendation	Rationale
Prior to app launch, host organisations should devise a publicity strategy to inform HCPs and the public of the app and encourage uptake. Continuously publicising the app over a longer period—and not just following app launch— may also serve to encourage app uptake	The Yellow Card app was launched in July 2015, and the highest number of app users registered around this time. The second-highest number of registrants occurred Sept 9–10, 2015, immediately following a brief update about the app on the gov.uk website [30]. Similarly, the Lareb app was launched at the end of January 2016. While the highest number of users registered immediately following the launch in February 2016, there was a second surge of app registrants in April following an article published in a Dutch nursing magazine [31]
The app should be hosted by an organisation that is trusted by and preferably familiar to its target users. If not familiar, steps should be taken to increase awareness of the host organisation in the peri-launch period. For example, the app could be promoted to HCPs via NCAs and/or professional bodies	A qualitative study found that respondents would be more likely to use an app that was associated with a familiar and trustworthy source [8]NCAs and professional bodies are preferred senders of risk information for professionals. Information about the app seems to be sufficiently related to that topic to apply the findings of the SCOPE survey [24]
Efforts to inform HCPs and patients about the app should focus on those who use other health apps. Hence, it is likely to be beneficial to advertise the app in other health apps	Patients and HCPs who already use other health apps were more interested in the app than those that did not use other health apps. It was also shown that younger patients tended to be more interested in the app [22]
Country-specific communication plans should be developed	Possible channels and sources for promoting the app will differ per country; therefore, country-specific strategies are needed. In the quantitative survey study, there were quite a few patients who had never heard of their national agency [22]

HCP healthcare professional, NCA national competent authority, SCOPE Strengthening Collaboration for Operating Pharmacovigilance in Europe

Mobile Application Impact and Miscellaneous Recommendations

The recommendations presented in Table 6 relate to the assessment of the overall impact of mobile reporting apps for PV and other stand-alone recommendations.

Table 6

Recommendations on assessment of the overall impact of applications (apps) for pharmacovigilance and other stand-alone recommendations

Recommendation	Rationale
The app should be considered an additional channel for ADR reporting	Analysis has shown very little qualitative difference between reports from ‘conventional means’ and reports from the app [25]
Consideration could be given to having the app tested by an independent reviewer that specialises in health apps prior to launch	Some countries have healthcare specific app stores that contain apps that have been tested by the relevant national health services and certified for usability, utility, and security. This can generate confidence in and increase awareness of the app
Consideration should be given to the means of measuring the public health impact of the app after launch with HCPs and patients	Measuring the impact of any mobile app can be challenging. It is easy to capture the number of downloads or reports, although these figures do not provide insights into the value of news feeds and safety data to patients or the value provided to HCPs and whether it impacts their prescribing. Valuable metrics may include: How often reports are sent How often the app is opened, how many new downloads, how many ADRs reported Impact of signals arising from the app compared to traditional reporting mechanisms

ADR adverse drug reaction, HCP healthcare professional

Discussion

A series of recommendations relevant to the development of mobile apps for the reporting of ADRs are presented with the intention of informing PV professionals and other stakeholders, particularly those with an interest in the development of research methods and digital technologies. Each recommendation is based on the experiences and outcomes of the research conducted under the auspices of IMI WEB-RADR. Collectively, the recommendations should not be considered a comprehensive treatise on the subject, rather they should be considered within the context of the entire corpus of research and regulatory guidance documents that exist outside of IMI WEB-RADR.

Since the start of the WEB-RADR project, mobile ADR-reporting apps have been launched in five separate countries. This group of participating countries has presented both diverse target populations of HCPs and non-HCP users, predominantly patients, as well as widely varying reporting systems and diverse technical resources that have impacted how each app has been configured. It is our hope that, having launched these apps in five unique settings, the recommendations, when taken together with their caveats, will be applicable to many other settings. That said, careful consideration should be given to the generalisability of the recommendations in circumstances that are substantially different from those described in the original research.

Overall, our experience has indicated that a deep understanding of the target app-user population, through local knowledge and potential user outreach prior to embarking on development activities, can facilitate effective design from the start. However, it is equally important to continue improving the app based on user feedback and evaluation of outputs after any public release—particularly following initial launch and major updates. Additionally, preferences and needs for patient safety information and reporting can vary greatly among different groups, and a balance must be struck between providing enough features that are useful to a wide range of users without attempting to create an app that ‘does it all’.

Communication in support of the launch of the app is important, followed by further provision of information at appropriate intervals to keep stakeholders informed about enhancements and new features of the software. Once established, a good place to inform patients about the app is at first prescription or first dispensing of a medicine. Patients are most likely to experience ADRs when starting new medications [32].

It is important to note that even if an app for ADR reporting is well-designed and user-friendly, it will not serve its purpose if the target population is not familiar with that purpose. Patient awareness of reporting ADRs is therefore critical to adoption and utilisation of the app and may be a factor beyond the scope of designing and developing the app itself. It might be necessary to raise public awareness of ADR reporting prior to evaluating the need for an app; in other cases, the existence of a mobile app could serve to drive awareness of ADR reporting among the public, and it may be useful to address underreporting barriers, as part of a publicity strategy. More research is needed to explore ways to encourage patient reporting of ADRs in a balanced way that does not lead to bias or soliciting.

Since release, each app has seen varying degrees of exposure. The Yellow Card mobile app was launched as an extension of MHRA’s Yellow Card Scheme—a safety reporting programme for human medicinal products that has been running for over 50 years. To date, MHRA has received 505 ADR reports through its mobile app, launched in July 2015 (see Table 1). Objectively speaking, this number pales in comparison to the 26,000 ADR reports the MHRA received in 2016 directly from HCPs and the public, and there is a tendency to view this as a shortcoming of the mobile app itself [33]. However, the value identified by users was much broader than just reporting and extended to engagement with the wider PV system and accessing timely safety information. As smartphone users increasingly use their mobile devices to access the Internet (in the UK, for example, 73% of adults browsed the Internet using their mobile phones in 2017, compared to 36% in 2011), so grows the opportunity for health authorities to reach patients through digital channels where they are already seeking information [34].

It is not practical to expect the outputs from novel, patient-centric reporting channels to mimic those from traditional channels, particularly as the value of patient-reported individual case safety reports (ICSRs) has only been recognised relatively recently [3]. There is therefore a need to re-calibrate expectations and perspectives when evaluating mechanisms for patient ADR reporting and establish a methodology that can allow an assessment of these initiatives entirely on their own, and not in the context of traditional reporting channels. For instance, Lareb has found that its Bijwerking app has encouraged participation from new members of the public, specifically HCPs, who have never submitted an ADR report previously, suggesting that the existence of the mobile app has increased the breadth and diversity of reporters [35]. Additionally, evaluation of reports originating from the apps revealed that while the resulting data had lower completeness scores, the app reports demonstrated clinical quality that did not differ significantly from that of reports submitted via conventional reporting channels and contributed to signal detection and safety evaluation activities.

Probably the most important recommendations are those that relate to engagement of users of the app, in particular, establishing the two-way interchange of safety information. Users of the app valued the provision of graphs and charts showing the safety profile of their medicines. This positive reinforcement of the value of transmitting ICSR was a feature of the feedback received during focus groups and interviews. However, a commitment to provide automated feedback should not override the current practices in terms of providing feedback for other reporting routes [22]. In general, provision of more detailed feedback information should be well thought through, as it may introduce questions or concerns relating to the complexity of mandatory ADR reporting requirements that may require time to address.

Equally important was the protection of data privacy, in accordance with national and European laws. Patients wanted assurance that the highest standards of confidentiality were maintained during the data reporting and transfer processes. Patients also found it important to be able to review their own previously submitted reports and to submit corrections or follow-up information later, if needed. Overall, the app and surrounding publicity brought the importance of PV to the attention of various stakeholders in public health. In the case of HCPs, this served as a reminder to report suspected ADRs, but for patients and caregivers, the app provided an alerting mechanism, highlighting the value of the public in recognising potential safety concerns.

The app could be considered an important mechanism to facilitate compliance with the European Union PV framework legislation which became operational in July 2012, obliging all NCAs and MAHs to record and report cases of suspected ADRs received from patients. Careful consultation with a range of patients and patient support groups helped in the design and delivery of a functional app to facilitate patient reporting. An app provides an additional means of reporting suspected ADRs and communication and, furthermore, projects an innovative and progressive image; in turn this fosters engagement with stakeholders. Health authorities might consider providing an app as an additional means of reporting and/or communication; however, the value versus opportunity costs, which were not specifically evaluated in WEB-RADR, are likely to vary from country to country. It is important that regulators are accessible through means that patients and HCPs use in their day-to-day lives; it should not be any more challenging to report an ADR than to use any other service. Furthermore, responsiveness to new technologies projects an innovative and progressive image that in turn fosters trust and improves dialogue with stakeholders.

A maintenance and support programme has been established with the aim of ensuring the sustainability of the app; however, questions remain about its long-term utility. From the patient perspective, there may be no need to repeatedly use the app to submit an ADR report; many patients will not experience multiple ADRs (if at all); thus, most users will not find it necessary to keep the app on their device, let alone use it regularly. Conversely, for HCPs, the app may prove useful on multiple occasions and could be of even greater utility if the app was integrated with other software, such as prescribing records, dispensing systems and other healthcare systems. The WEB-RADR project has established a follow-up programme to develop the app using additional medical ontology mappings and by evaluating additional application programming interfaces for linking to other software.

In the future, the potential value of the app in supporting the notification of product quality issues and/or product complaints should be investigated. This could yield important information about manufacturing issues or batch-related concerns, particularly if used in conjunction with built-in image capture technologies. Similarly, specialist use of the app should be considered if an emerging safety issue arises with a product or class of medicines. Any situation such as this would require the close monitoring of a cohort of patients in a defined setting, and the app could provide a useful adjunct to traditional methods of PV by providing safety data in real time for rapid evaluation.

Conclusions

Over a period of 3 years, IMI WEB-RADR has addressed several important research questions relevant to the development of an app for reporting suspected ADRs. The resultant recommendations point to a series of pragmatic steps that those working in the PV community should consider before designing and building an app in support of safety reporting. The recommendations support the idea that the app is an important adjunct to existing ADR reporting pathways; however, engagement of app users is key to successful app development and adoption, in particular, in establishing the two-way interchange of safety information. As a younger generation of app-literate patients matures, it seems likely that app-based reporting will grow in importance and that, at some point, structured electronic reports will supplant the use of traditional paper forms.

A mobile application (app) designed for adverse drug reaction (ADR) reporting and product safety alerts can help to augment pharmacovigilance activities and extend a health authority’s reach to patients and healthcare professionals.

Of particular value to app users was the ability to learn about the safety profiles of medicines through user-friendly, interactive graphics within the app as well as privacy and data protection features.

While uptake and use of the app to date seems modest in comparison with other ADR-reporting modalities, it is reasonable to expect that app-based reporting will grow in importance as a younger generation of app-literate patients matures and smartphone owners increasingly use their mobile devices to access the Internet.