Recommendations for developing a lifecycle, multidimensional assessment framework for mobile medical apps.

Digital health and mobile medical apps (MMAs) have shown great promise in transforming health care, but their adoption in clinical care has been unsatisfactory, and regulatory guidance and coverage decisions have been lacking or incomplete. A multidimensional assessment framework for regulatory, policymaking, health technology assessment, and coverage purposes based on the MMA lifecycle is needed. A targeted review of relevant policy documents from international sources was conducted to map current MMA assessment frameworks, to formulate 10 recommendations, subsequently shared amongst an expert panel of key stakeholders. Recommendations go beyond economic dimensions such as cost and economic evaluation and also include MMA development and update, classification and evidentiary requirements, performance and maintenance monitoring, usability testing, clinical evidence requirements, safety and security, equity considerations, organizational assessment, and additional outcome domains (patient empowerment and environmental impact). The COVID-19 pandemic greatly expanded the use of MMAs, but temporary policies governing their use and oversight need consolidation through well-developed frameworks to support decision-makers, producers and introduction into clinical care processes, especially in light of the strong international, cross-border character of MMAs, the new EU medical device and health technology assessment regulations, and the Next Generation EU funding earmarked for health digitalization.

INTRODUCTION

The COVID‐19 pandemic has unquestionably accelerated the transition to digital health (DH) (Pandey & Pal, 2020; Petracca et al., 2020). More specifically, mobile apps have given health care systems the opportunity to support public health surveillance during the most hard‐hit periods, at once emphasizing the need to picture and build cutting‐edge care models for the upcoming future.

The promise of mobile apps to manage individual health created growing expectations over time, leveraging on the value of personalized data captured outside of traditional hospital and clinical settings (Ku & Sim, 2021). However, their actual adoption in clinical care was still unsatisfactory before the pandemic. While the uptake of DH solutions during the pandemic was widespread and shaped by the organizational and institutional systems in which it took place (Petracca et al., 2020), it was largely favored by temporary, flexible policies that predominantly affected low‐risk technologies with high potential for patients (Kadakia et al., 2020). For example, the United States (US) Food and Drug Administration (FDA) issued temporary guidance to expand access to digital therapeutics for psychiatric disorders without the submission of a pre‐market notification (FDA, 2020e).

International agencies have discussed the need for regulatory guidelines to classify mobile health apps and detail their pre‐market route. However, since many mobile apps at present are not considered medical devices, they have largely escaped oversight, resulting in calls for expanded regulation (Kasperbauer & Wright, 2020). Similarly, for complex digital technologies, such as those based on artificial intelligence and machine learning, little regulatory guidance is available (Muehlematter et al., 2021).

In addition, where regulatory frameworks exist, they have rarely been followed by formal reimbursement or coverage arrangements. Likewise, there is no common framework for assessment purposes in the scholarly debate, with those available often proposing generic assessment models for all digital services, which fail to properly weigh the specifics of each single digital technology. Most of the contributions are merely rating tools aimed at supporting patients and clinicians in choosing the best option, often a mobile health app (Hensher et al., 2021; Lagan et al., 2020; Lagan et al., 2021). In fact, a systematic review of the scientific literature showed that available evaluation frameworks for mobile apps are not suitable for use in health technology assessment (HTA), as none cover all of the HTA core domains (Moshi et al., 2018). As a consequence, it is still unclear how to disseminate appropriate app interventions to patients and providers (Leigh & Ashall‐Payne, 2019), despite having amply identified strategies and practicalities for their thorough diffusion and integration into routine clinical practice (Gordon et al., 2020). Without full integration into the care continuum alongside traditional technologies and interventions, DH fails to reach its highest possible value (Gordon et al., 2020). For mobile apps to make the leap to routine use for clinical purposes, regulators and agencies need to ensure a robust pathway for their adoption, embracing the universal paradigm of value to enable an unbiased comparison with other health care technologies competing for the same resources.

Herein, we focus on mHealth apps within the digital medicine (DM) area, namely mobile medical apps (MMAs), defined by the FDA as mobile apps that incorporate device software functionalities (FDA, 2019b). This allows for simultaneously providing a technological specification—that of software accessed through specific platforms such as smartphones and tablets—and a narrowing of attention from DH to DM, as such excluding wellness and support products not requiring the same degree of regulatory oversight. MMAs also include digital therapeutics (DTx) accessible through mobile platforms such as smartphone and tablets, as well as artificial intelligence and/or machine learning (AI/ML)‐based medical apps (see Box 1 for definitions of DH solutions).

Like integrated approaches to decision‐making already proposed for medical devices (Tarricone et al., 2020), a dynamic approach that looks at the entire lifecycle of digital technologies and takes into account their distinctive features is needed (Tarricone et al., 2021) (i.e., one that could foster a participatory approach and surpass the sharp distinction between the pre‐market and post‐market stages). As for medical devices (MDs), also the MMAs lifecycle consists of four main stages: the (i) pre‐clinical, pre‐market; (ii) clinical, pre‐market; (iii) diffusion, post‐market; and (iv) obsolescence & replacement. Similarly to MDs, the lifecycle approach for MMAs, to be effective, requires anticipation of what evidence will be needed at each stage (Tarricone et al., 2020). What matters in the MMAs case are the type of evidence and the length of the stages that may differ from conventional MDs thus making their assessment even more challenging. Experimental studies are – for instance ‐ even more widely inapt for MMAs, that can evolve very rapidly, incorporate continuous measurement of the intervention effects, and need to be constantly readjusted just to keep up with their intended use. This consideration also affects the obsolescence stage that, for MMAs, is easier to be replaced by software adaption for instance. Moreover, retention is one of the most significant issues for any MMAs, as only small portions of users show consistent access to mHealth apps over time (Tarricone et al., 2021).

Given this lack of comprehensive guidance on the evaluation of MMAs and the concurrent need to provide thorough evidence to sustain their deployment, we aim at providing principles and recommendations for developing a multidimensional (e.g., clinical, economic, usability, equity, etc. dimensions) assessment framework to: (i) address all relevant domains that should be considered in the assessment of MMAs, (ii) leverage on the specific features of MMAs, (iii) be compatible with regulatory and HTA purposes, and (iv) inform policy decisions on MMAs.

METHODS

In developing a multidimensional framework for the assessment of MMAs, with a view to support policy decisions on their sustained uptake and diffusion within healthcare systems, we followed a multistep approach.

Policy document review

First, with the aim of mapping the current state of the art in terms of MMA assessment from a regulatory and coverage or reimbursement point of view, we conducted a targeted or a state‐of‐the‐art review (Grant & Booth, 2009) of the most relevant policy documents published by international organizations (i.e., the World Health Organization [WHO] and the International Medical Device Regulators Forum [IMDRF], a voluntary network of 10 jurisdictions that includes the US and the European Union [EU] and seeks to harmonize medical device assessment), supranational institutions (such as the EU), single member states, and the US. In this mapping, the FDA in the US was explicitly included as a leading regulatory organization, responsible for ensuring the safety, efficacy, and security of drugs, biological products, and medical devices, and as a pioneer institution in embracing the lifecycle logic in the evaluation of health care technologies. Public sources, newsletters, and websites were regularly searched for regulatory or HTA guidelines specifically dedicated to MMAs throughout the COMED project, from March 2018 to May 2021, complemented by snowballing and inclusion of documents already known to the research team and published in English.

Expert focus group

The results of the review, along with the identification of the main features that distinguish MMAs from other health care technologies (Tarricone et al., 2021), provided the basis for preliminary development of an initial set of recommendations, which were subsequently discussed within an international focus group comprised of key relevant stakeholders to solicit their input and feedback. Eight participants represented all key stakeholder groups (i.e., industry, patient associations, HTA agencies, policy makers, and academia) (Table A1).

The focus group methodology was selected as a suitable means to collect feedback and solicit input due to: i) its flexibility, ii) the ability to directly interact with domain experts and probe them on key design ideas, iii) the ability to collect both qualitative and quantitative data, iv) the fact that valuable information is created in the interaction between participants (Krueger & Casey, 2014).

Initially proposed as a 1‐day, in‐person workshop, the March 30, 2020, meeting was shortened to a half‐day online workshop due to COVID‐19 restrictions. Participants were presented with slides, summarizing the distinguishing features of MMAs and the proposed recommendations for their assessment, followed by discussion sessions, divided into two main parts: first, addressing the relevant domains that should be considered in assessing MMAs, in line with their specific features, and second, discussing each of the 10 recommendations identified by the research team.

Participants shared their views and suggested modifications to refine and improve the assessment framework. Based upon the panel's input, the set of recommendations was finalized and eventually shared within the panel.

RESULTS

Assessment of MMAs from a policy point of view: State of the art

In this section, we comparatively analyze the assessment methods and procedures for MMAs adopted by key international organizations such as the IMDRF and the WHO, jurisdictions such as the US and the EU, and other countries such as the United Kingdom (UK), France, and Germany that—during the COMED project—have issued frameworks or guidance documents relevant for our purposes (Tables 1, 2 and 3) in English.

TABLE 1

Target focus of regulatory and assessment approaches across jurisdictions

	IMDRF	WHO	EU	FRANCE	GERMANY	USA	UK
Organization	IMDRF	WHO	EC	HAS	BfArM	FDA	MHRA	NICE,
								NHSX
Purpose	Regulatory approval	Assessment for Decision‐Making	Regulatory approval	Assessment for Coverage and Reimbursement Decisions	Assessment for Coverage and Reimbursement Decisions	Regulatory approval	Regulatory approval	Assessment for Coverage and Procurement Decisions
Target Object	Software as Medical Device (SaMD)	Digital Health (DH)	Medical Device Software (MDS)	Health Apps and Smart Devices (mHealth)	Digital Health Application (DIGA)	Device Software Functions (DSF), including Mobile Medical app (MMA)	Medical Device Standalone Software	Digital Health Technologies (DHT)
Definition	Software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device	The use of digital, mobile, and wireless technologies to support the achievement of health objectives.	Software that is intended to be used, alone or in combination, for a purpose as specified in the definition of a “medical device” in the Medical Device Regulation (MDR) or In Vitro Diagnostics Regulation (IVDR), regardless of whether the software is independent or driving or influencing the use of a device	Medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices	EU MDR Class I or IIa medical devices achieving their medical purpose through core digital functions and used either by patients alone or patients with healthcare professionals.	All SAMD and SIMD that are not used for administrative support of a healthcare facility, for maintaining or encouraging a healthy lifestyle, to serve as electronic patient records, or for transferring, storing, converting formats, or displaying data	Software not incorporated in a device possessing its own medical purpose	Apps, programmes and software used in the health and care system that may be standalone or combined with other products such as medical devices or diagnostic tests
				Smart devices are devices connected to the internet that can collect, store, process and send data or that can take specific actions based on information received
		Digital health is inclusive of both mHealth and eHealth
Main Reference Documents	Software as a Medical Device (SaMD): Key Definitions (IMDRF, 2014)	Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment (WHO, 2016)	Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR (European Union, 2017a, 2017b)	Good Practice Guidelines on Health Apps and Smart Devices (Mobile Health or mHealth) (Haute Authorité de Santé, 2016)	The Fast‐Track Process for Digital Health Applications (DiGA) according to Section 139e SGB V: A Guide for Manufacturers, Service Providers and Users (BfArM, 2020b)	Policy for Device Software Functions and Mobile Medical Applications (FDA, 2019b)	Guidance: Medical device stand‐alone software including apps (including IVDMDs) (Medicine and Healthcare Products Regulatory Agency, 2021)	Evidence Standards Framework for Digital Health Technologies (NICE, 2018)

TABLE 2

Risk‐categorization models developed across jurisdictions

Jurisdictions	IMDRF	WHO	EU	FRANCE	GERMANY	USA	UK
Main reference documents	“Software as a Medical Device": Possible Framework for Risk Categorization and Corresponding Considerations (IMDRF, 2014)	Classification of digital health interventions v1.0. A shared language to describe the uses of digital technology for health (WHO, 2018)	Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR (European Union, 2017a, 2017b)	Good Practice Guidelines on Health Apps and Smart Devices (Mobile Health or mHealth) (Haute Authorité de Santé, 2016)	The Fast‐Track Process for Digital Health Applications (DiGA) according to Section 139e SGB V. A Guide for Manufacturers, Service Providers and Users (BfArM, 2020b)	Developing a Software Pre‐ Certification Program: A Working Model (FDA, 2019b)	Evidence Standards Framework for Digital Health Technologies (NICE, 2018, 2021)
Risk categorization	Two‐dimensional: significance of the information provided by the SaMD to the healthcare decision state of the healthcare situation or condition	DH is classified according to the different ways in which DH interventions are being used to support health system needs and not on risk level	Two‐dimensional: significance of the information provided by the SaMD to the healthcare decision state of the healthcare situation or condition	Two‐dimensional: main target user:‐ general public ‐ patient/carers/family/patient associations ‐ healthcare professionals directly with their patients ‐ healthcare professionals directly with their peers (e.g., teamwork) main intended use:‐ information, general advice ‐ primary prevention, health promotion, manual data entry and acquisition without analysis ‐ secondary and tertiary prevention, therapeutic patient education ‐ analysis of data, medical evaluation contributing to assessment, diagnosis, monitoring throughout the care pathway, impact on treatment	Two‐dimensional: significance of the information provided by the SaMD to the healthcare decision state of the healthcare situation or condition	Two‐dimensional: significance of the information provided by the SaMD to the healthcare decision state of the healthcare situation or condition	One‐dimensional: (1) Detailed functional use
Risk classes	Categories I, II, III, IV	DH are classified around the target audience:(1) Clients (2) Healthcare providers (3) Health systems managers (4) Data services	Classes I, IIa, IIb, III (and A, B, C, and D for IVD). Rule 11 has been introduced to address the risks related to the information provided by Medical Device Software (MDSW) and is divided into three sub‐rules (an analogous reasoning is applied to IVDR MDSW with classes A, C, D):‐ 11a: intended to provide information which is used to take decisions with diagnostic or therapeutic purposes (in this case MDSW is classified as class IIa, except if such decisions have an impact that may cause death or an irreversible deterioration of a person's state of health, in which case it is in class III; or a serious deterioration of a person's state of health or a surgical intervention, in which case it is classified as class IIb); ‐ 11b: intended to monitor physiological processes or parameters (in this case MDSW is classified as class IIa, except if it is intended for monitoring of vital physiological parameters, where the nature of variations of those parameters is such that it could result in immediate danger to the patient, in which case it is classified as class IIb); ‐ 11c: all other uses (in this case it is classified as class I).	Low Criticality, Medium Criticality, High Criticality	Classes I and IIa	Categories I, II, III, IV	10 Classes grouped in Evidence Tiers A, B, C.• Tier A: system impact • system services: DHTs with no measurable patient outcomes but which provide services to the health and social care system • Tier B: understanding and communicating • inform: provides information, resources or activities to the public, patients or clinicians; includes information about a condition or general health and lifestyle • health diaries: includes general health monitoring using fitness wearables and simple symptom diaries • communicate: allows 2‐way communication between citizens, patients or healthcare professionals • Tier C: interventions • preventative behavior change: address public health issues like smoking, eating, alcohol, sexual health, sleeping and exercise • self‐manage: allows people to self‐manage a specified condition; may include behavior change techniques • treat: provides treatment; guides treatment • active monitoring: tracking patient location, using wearables to measure, record or transmit data (or both) about a specified condition; uses data to guide care • calculate: a calculator that impacts on treatment, diagnosis or care diagnose: diagnoses a specified condition; guides diagnoses.

TABLE 3

Assessment approaches across jurisdictions

Jurisdictions	IMDRF	WHO	EU	FRANCE	GERMANY	USA	UK
Organization	IMDRF	WHO	EC	HAS	BfArM	FDA	NICE
							NHSX
Main reference documents	Software as a Medical Device (SaMD): Clinical Evaluation (IMDRF, 2017)	Classification of digital health interventions v1.0. A shared language to describe the uses of digital technology for health (WHO, 2018)	Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR (Medical Device Coordination Group, 2019)	Good Practice Guidelines on Health Apps and Smart Devices (Mobile Health or mHealth) (Haute Authorité de Santé, 2016)	The Fast‐Track Process for Digital Health Applications (DiGA) according to Section 139e SGB V: A Guide for Manufacturers, Service Providers and Users (BfArM, 2020b)	Developing a Software Pre‐Certification Program: A Working Model (FDA, 2019b)	Evidence Standards Framework for Digital Health Technologies (ESFDHT) (NICE, 2018, 2021)
	Software as a Medical Device (SaMD): Application of Quality Management System (IMDRF, 2015)						Digital Technology Assessment Criteria for Health and Social Care (DTAC) (NHSX, 2021)
			Guidance on Clinical Evaluation (MDR)/Performance Evaluation (IVDR) of Medical Device Software (Medical Device Coordination Group, 2020)
Target focus	Product	Product	Product	Product	Product	Product & Firm	Product
Assessment approach	Clinical Evaluation Model embedded in SaMD Realization and Use Processes	Lifecycle Approach	Continuous and iterative clinical (MDR)/performance (IVDR) evaluation embedded in the QMS (IMDRF inspired)	Step‐Wise Evaluation model based on:	Fast‐track process for the assessment of DiGA, focusing on product quality and evidence underlining positive healthcare effects	Total Product Lifecycle (TPLC) Approach for continued evaluation from pre‐market development to post‐market performance	Risk‐based model for evaluation, with cumulative evidence requirements for demonstrating effectiveness and economic impact
Assessment domains/steps	(1) Clinical Validity (2) Analytical Validation (3) Clinical Validation	Monitoring:‐ Functionality, i.e., the ability to support the desired intervention; ‐ Stability, i.e., the ability to remain functional both under normal and anticipated peak conditions for data loads. ‐ Fidelity, i.e., whether the intervention is delivered as intended from both a technical and user perspective. Monitoring fidelity consists of three different categories: a) monitoring the technical fidelity in the implementation process; b) monitoring external barriers that might cause the system not to function as expected; c) monitoring the compliance of the digital health system users; ‐ Quality, i.e., the overall quality of the intervention in terms of excellence, values, conformance, fitness for purpose and ability to meet or exceed expectation. Two main aspects should be addressed when monitoring quality. The first one pertains to user capabilities, to guarantee users do enter information accurately and use the system correctly overall. The second dimension relates to the overall quality of the intervention as a prerequisite for the effectiveness of the intervention Evaluation:‐ Feasibility: Assess whether the digital health system works as intended in a given context; ‐ Usability: Assess whether the digital health system is used as intended; ‐ Efficacy: Assess whether the digital health intervention achieves the intended results in a research (controlled) setting; ‐ Effectiveness: Assess whether the digital health intervention achieves the intended results in a non‐research (uncontrolled) setting; ‐ Implementation research: Assess the uptake, institutionalization, and sustainability of evidence‐based digital health interventions in a given context, including policies and practices	(1) Planning Evidence Generation (2) Demonstrating Clinical association to the outcomes (3) Updating Clinical Evaluation through Post‐Marketing Clinical Follow‐Up	(1) Informing User:‐ Description ‐ Consent (2) Health Content:‐ Design of initial content ‐ Standardization ‐ Generated content ‐ Interpreted content (3) Technical Content:‐ Technical design ‐ Data flow (4) Security/Reliability:‐ Cybersecurity ‐ Reliability ‐ Confidentiality (5) Usability/Use:‐ Usability/design ‐ Acceptability ‐ Integration/import	(1) Mandated Requirements:‐ Security ‐ Functionality ‐ Quality ‐ Data Protection ‐ Data Security ‐ Interoperability (2) Positive Care Effects:‐ Medical benefits ‐ patient‐relevant structural and procedural improvements If the assessment process is positive the product is included in the DiGA Directory, a repository of apps available for prescription	(1) Excellence appraisal (2) Review Pathway Determination (3) Streamlined Pre‐Market Review (4) Real World Performance The Excellence Appraisal introduces a firm‐based assessment approach based upon: (1) Product Quality (2) Patient Safety (3) Clinical responsibility (4) Proactive Culture	ESF:(1) Clinical Effectiveness (2) Economic Impact DTAC:(1) Clinical safety (2) Data protection (3) Technical assurance (4) Interoperability (5) Usability and accessibility
					Type of Recommendations:(1) Permanent acceptance (2) Preliminary Acceptance
Economic domains	N/A	N/A	N/A	N/A	N/A	N/A	Economic impact relative to the financial risk:(1) key economic information (2) appropriate economic analysis (3) economic analysis reporting standards
Study Designs	N/A	Hybrid approaches	Prospective study may be required for higher risk MDSW, whereas retrospective analyses may be sufficient for lower risk ones	N/A	Minimum requirement of one retrospective comparative study (although prospective ones are more appreciated) and must be conducted, at least partially, in Germany Recognized relevance of alternative study designs (e.g. PCT, SMART, or MOST)	N/A	Tier C: High quality observational or quasi‐experimental studies (minimum), High quality (quasi‐) experimental intervention studies (best practice), High quality RCT or meta‐analysis of RCTs (best practice)
Real World Data	Clinical evaluation should be intended as a dynamic summary that changes through continuous RWP learning	Clinical evaluation must be generated as a blend effectiveness and implementation trial elements	Manufacturers are asked to continuously monitor the device safety, effectiveness and performance	N/A	Declared preference for RWD based on healthcare practice	RWP Analytics Framework(1) Real World Health analytics (2) User Experience Analytics (3) Product Performance Analytics	Request for ongoing data collection to show DHT usage and value
Reference to AI/ML technologies	IMDRF Artificial Intelligence Medical Devices (AIMD) Work Item (IMDRF, 2021)	N/A	The White paper on Artificial Intelligence is a non‐industry specific document that discusses the adoption of AI across sectors and activities (EC, 2020).	N/A	N/A	Artificial Intelligence and Machine Learning in Software as a Medical Device (FDA, 2021)	An additional Evidence Framework is expected as from the collaboration between the NICE and MHRA
			The most relevant aspect for SaMD is that of suggesting a conformity assessment for high‐risk AI applications			Two principles: ‐ SaMD Pre‐Specifications ‐ Algorithm Change Protocol Two ex‐novo regulatory pathways:‐ Document Approach ‐ Focused FDA Review Pathway
	Work in progress

Assessment frameworks: Aims and targets

First, it is important to note that the assessment frameworks developed by these jurisdictions (i.e., IMDRF, FDA, European Commission [EC], and MHRA) have different purposes (i.e., in some cases, the goal is to regulate market access of DH, whilst in others, it is to help decide upon coverage and/or reimbursement/procurement) (Table 1). As to the target object, the guidance documents cover broad definitions of DH, ranging from the WHO definition, which refers to the use of digital, mobile, and wireless technologies to support the achievement of health objectives, to the more specific definition adopted by Federal Institute for Drugs and Medical Devices in Germany (Bundesinstitut für Arzneimittel und Medizinprodukte, BfArM), which focuses on the EU Medical Device Regulation (MDR) Class I or IIa medical devices achieving their medical purpose through core digital functions, used either by patients alone or patients with health care professionals. However, in all cases, the critical aspect is the link between the software and its impact on health, without which the digital solution ceases to be the target of the assessment framework.

Classification criteria

The classification of DH tends to be risk‐based, inspired by the work done by the IMDRF. In 2014, the IMDRF defined the concept of Software as Medical Device (SaMD) and grouped them according to the level of risk into four classes (IMDRF, 2014) (Table 2). A different approach was adopted by the National Institute for Care and Health Excellence (NICE) in the UK that lists 10 classes of apps, software, and programs grouped in three evidence tiers (NICE, 2018, 2021) (Table 2). The only exception to the risk‐based classification is that of the WHO, which, instead, identifies a taxonomy based on the different ways in which digital and mobile technologies are being used to support health system needs (WHO, 2018) (Table 2).

Assessment approaches

Major differences can, however, be found in assessment approaches and domains (Table 3). As to the approach, the majority of frameworks advise some form of iterative process but differ in how they are formulated, ranging from a linear, step‐wize method to a clockwise total product lifecycle from the US FDA. The FDA arguably represents the most innovative regulation system and focuses on, differently from other jurisdictions, both the product and the manufacturer, witnessed by the launch of the Digital Health Center of Excellence (DHCoE) (FDA, 2020b) within the Center for Devices and Radiological Health (CDRH), aimed at providing regulatory advice and support to the FDA's regulatory review of DH technology. The establishment of the DHCoE is consistent with the Digital Health Innovation Plan (FDA, 2020c) that aims—among others—to foster FDA's strengths and expertize in CDRH's DH unit and re‐imagine the assessment process of device software functions. For this purpose, the Software Pre‐Certification Program (or Pre‐Cert program) (FDA, 2019a) was conceived with the aim of providing either pre‐market review exemptions to lower risk technologies or a streamlined pathway for higher‐risk products to foster system efficiency (Table 3). The first step of the Pre‐Cert program is the excellence appraisal, which represents the essence of this “firm‐based” approach, whereby developers are audited and pre‐certified as long as they “have demonstrated a robust culture of quality and organizational excellence (CQOE) and are committed to monitoring real‐world performance” (FDA, 2019a). The CQOE assessment is based on demonstration of five excellence principles (Table 1), and each domain is measured against a set of key performance indicators (KPIs). If deemed eligible, organizations receive either Pre‐Cert levels L1 or L2. (Level 1 Pre‐Cert is granted in case of objective demonstration of excellence across all five principles but with limited track record in the development, delivery, and maintenance of products, whereas Level 2 Pre‐Cert is granted to those with more extensive experience). In the second step, the results of the excellence appraisal (i.e., recognition of L1 or L2 status) are coupled with an IMDRF‐inspired risk categorization framework for the sake of review pathway determination. In general, lower‐risk devices will be directly marketed, whereas higher‐risk devices will have to undergo the third step of a streamlined pre‐market review. Finally, both the product and the developer are monitored throughout the fourth step, real‐world performance, which aims at verifying ongoing excellence and generation of clinical evidence through post‐market data. In 2019, a pilot test was launched with 9 companies of varying sizes, age, and organizational structure, the findings of which will help inform large‐scale beta testing. As of 2020, the pilot has allowed to iteratively test different methods and approaches through mock excellence appraisals, aggregating results in a library of activities, processes, and KPIs used by high‐performing organizations. Preliminary results also helped understand how to operationalize the real‐world performance domain, particularly regarding user experience and product performance analytics, but further real‐world health analytics investigation is still needed. Measurement of health benefits has also proven to be challenging, and additional work to identify these measures is needed (FDA, 2020a).

A similar commitment toward establishing ad hoc approaches for DH can be found in the EU, where a number of policy cooperation platforms have also been established, such as the eHealth Network, focusing on interoperability and standardization; the eHealth Action (eHAction), providing technical and scientific advice to the network; the eHealth Stakeholder Group (eHSG), contributing to the development of eHealth policy; and the Joint Action for the European Health Data Space (TEHDAS), supporting the EC plan for the European Health Data Space (EHDS). However, the assessment approach of mHealth apps does not truly differentiate from the one introduced for conventional medical devices by the new EU MDR (EU 2017/745), implemented in May 2021, and In Vitro Diagnostics (IVDR) (EU 2017/746) to be implemented in May 2022 (Table 1), which suggest drafting an iterative clinical development plan across the pre‐ and post‐market phases (Table 3).

Domains of assessment

As to the domains' target of assessment, the jurisdictions vastly differ. The clinical dimension is present in all frameworks and mainly pertains to assessment of the product's impact on users' health. However, besides the general indication of proportionality with the level of risk, few recommendations can be found as to how to assess the clinical impact. The WHO vaguely refers to hybrid approaches, by blending effectiveness and implementation trial elements (Table 3). The German Digital Health Application (DiGA) Guide (BfArM, 2020b) is the most precise and sets a preference for (possibly domestically generated) real‐world data based on health care practice rather than trial settings, as demanded by the interactivity of MMAs with their surrounding environment (Table 3). In proving positive care effects, although prospective comparative studies are more appreciated as they provide higher‐level evidence, the minimum requirement is set at the submission of at least one retrospective comparative study (e.g., case‐control studies, retrospective cohort studies, or intra‐individual comparisons). The guide also recognizes the relevance of alternative study designs and methods such as pragmatic clinical trials (PCT), sequential multiple assignment randomized trial (SMART), or multiphase optimization strategy (MOST). Nonetheless, a converged and established approach for assessing the quality of the study design, especially with the respect to the selected comparator, has yet to be fully defined.

Other domains pertain to the technical contents, security and reliability, and usability, and are found, although under different definitions and with various levels of details, in most frameworks. The economic dimension is present only in the UK, where NICE published, in March 2019, its “Evidence Standards Framework (ESF) For Digital Health Technologies (DHTs)”; it was further updated in April 2021 (NICE, 2018, 2021). The ESF describes standards for the evidence that should be available, or developed, for DHTs to demonstrate their value in the UK health and social care system. This includes evidence of economic impact relative to financial risk. The NICE approach tries to provide a comprehensive guide for developers on how to build a strong economic case, as was done for innovative medical devices and diagnostics (Campbell et al., 2018). As a way of informing decision‐making and investment decisions, NICE has identified three building blocks: (1) key economic information (i.e., user population size, care pathways—existing and proposed, and parameters for the economic model such as intervention, cost, resource use, and utilities), (2) appropriate economic analysis (i.e., budget impact analysis, cost‐consequence analysis, and cost‐utility analysis), and (3) economic analysis reporting seven standards (i.e., economic perspective, time horizon, discounting, sensitivity analyses, equity analyses, additional analytical methods, and critique of the economic analysis) (Table 3). A separate framework is also provided by the Digital Technology Assessment Criteria for Health and Social Care (DTAC), published by the NHSX—a joint unit of National Health Service (NHS) England and the Department of Health and Social Care—to provide clearer direction on both the development and procurement of sound DHT (NHSX, 2021). This is to be understood as a broader and more practical counterpart to the more theoretical ESF by NICE and provides a measurement of the overall success of the assessment of the product or service and adherence to NHS service standards (Table 3).

Coverage and reimbursement

As to coverage and reimbursement decisions, the French National Committee for the evaluation of medical devices and health technologies (CNEDIMTS) assesses the added value of the app similarly to that of any other health technologies. For example, the MOOVCARE POUMON application (intended for medical telemonitoring of relapse and complications in patients with non‐progressive lung cancer), upon positive evaluation, was included in 2020 in the French List of Products and Healthcare Services Qualifying for Reimbursement (LPPR) with Added Clinical Value (ASA) level III (moderate improvement) compared to conventional care (i.e., monitoring by imaging and medical face‐to‐face consultations) (MedTech Reimbursement Consulting, 2020).

In a similar fashion, DiGA may be granted permanent or preliminary acceptance (i.e., when all requirements have been fulfilled but the positive care effect is not definitely proven). In this case, the company is granted a 12‐month (or exceptionally 24‐month) period to deliver the needed supplementary data and be inserted in the DiGA Directory, an online platform, similar to the NHS app library and to the French LPPR, intended for physicians and psychotherapists but also patients and their caregivers. Exhaustive information, such as the current status of evidence of the approved technologies, is provided through different levels according to the targeted audience, fostering transparency and allowing for more informed decisions. The DiGA Directory, hence, substantially provides an updated pool of apps that are available for prescription and reimbursable at the price set through negotiation between manufacturers and the Central Federal Association of the Health Insurance Funds.

Artificial intelligence‐ and machine learning‐based DH technologies

One last aspect concerns the consideration of AI/ML‐based DH products. Although several jurisdictions recognize the importance of developing an ad hoc framework, only the FDA currently offers guidance on the AI/ML in software as a medical device (SaMD)(FDA, 2021) (Table 3). The framework proposes the creation of a “predetermined change control plan” that includes “the types of anticipated modifications—“Software as a Medical Device Pre‐ Specifications”—based on the retraining and model update strategy, and the associated methodology—Algorithm Change Protocol—being used to implement those changes in a controlled manner that manages risks to patients.” Two important principles are hence recognized. SaMD pre‐specifications (SPS) are the pre‐envisioned types of changes the developer plans to achieve through real‐world learning in terms of SaMD performance, inputs, or intended use of AI/ML‐based SaMD. The algorithm change protocol (ACP) delineates the steps that should be followed to ensure that the modification does not hinder device safety and effectiveness, accounting for four components: data management, re‐training, performance evaluation, and procedures update. Conditional on the type of modification, three regulatory outcomes are possible. First, when the risk to patients of the modification is low, developers would simply have to document the modification and the analysis in the risk management and 510(k) files. Second, a focused FDA review when the technologies can be refined based on real‐world learning and training but still within the same intended use. Third, a new 510(k) for pre‐market review when none of the previous exemptions apply.

Recommendations for developing a lifecycle, multidimensional assessment framework for MMAs

The review of the policy documents highlighted how several jurisdictions and international organizations are keen to govern the growing introduction of DH technologies and MMAs through issuing and repeatedly updating regulatory and/or guidance documents aimed at assessing digital technologies before or after they enter routine practice. However, as noted by the international focus group participants, a systematic, comprehensive framework that could serve diverse purposes ranging from regulation of market authorization to purchasing and appropriate and safe use of MMAs is lacking. Analysis of available guidance and the focus group discussion were used to identify a list of recommendations that should be considered in the development of an assessment framework for MMAs adopting a lifecycle approach to evidence generation and assessment (Table 4).

TABLE 4

Multi‐dimensional assessment framework of mobile medical apps (MMAs). The DECALOGUE

Domain	Recommendation
MMA development and update	Shared decision‐making approaches to the development of apps can enhance the replicability of apps and ultimately lead to improved outcomes
Classification and evidentiary requirements	An unambiguous classification of MMAs should be associated with corresponding evidence generation requirements and the possibility to flexibly review the associated level of risk of every single MMA
Performance and maintenance monitoring	MMA manufacturers must ensure that technical system implementation does not threaten the overall effectiveness. Analytical validity must be determined by the manufacturers and a core set of indicators should be developed that are coherent with the MMA classification
Usability testing	Usability should be continuously monitored, both in the development phase of the solution and after its implementation in the field
Additional outcome domains	Besides conventional outcomes similar to all healthcare technologies, patient empowerment associated with use and potential environmental impact are distinguishing outcome domains of MMAs that, where relevant, should be appropriately measured and valued
Clinical evidence requirements	Flexible study designs that account for the specific characteristics of MMAs and can generate fast and efficient results should be adopted and coupled with flexible policy arrangements, based on the level of risk and the position in the product lifecycle of the app
Safety and security	The assessment of the risks related to data privacy, cyber security and misinformation and their potential impact are emphasized issues for MMAs
Equity considerations	Given the context of escalating inequalities and persisting technological divide, it is paramount to evaluate the net effect brought by MMAs on equity
Organizational assessment	The assessment of direct and indirect implications of the adoption of MMAs on the organizational level should focus on process, people, structure, culture or management impacts
Cost and economic evaluation	The evaluation of cost and economic impacts of MMAs should require no innovative forms and methodologies (reporting should follow established guidelines for economic evaluations), but rather new metrics for outcomes and different structures for cost

Recommendation 1. MMA development: Shared decision‐making approaches to the development of apps can enhance their replicability and ultimately lead to improved outcomes

The pre‐deployment phase of MMAs starts from their development in terms of content and features embedded in the app. Development factors are multiple and typically include the adoption of health behavioral theories in the app design process, as well as user and health care professional involvement (Adu et al., 2018). Engaging with end‐users and following robust theories could enable a more appropriate app design, as well as reduce the ineffective use of the app (Farao et al., 2020; McCurdie et al., 2012; Schnall et al., 2016). An increasing number of design methods for MMAs now exists, with different approaches to user needs and the establishment of user models (personas) as archetypes of groupings based on behaviors, attitudes, and goals to guide the functional design (Duan et al., 2020). Particularly for lower‐risk apps that may never undergo a formal clinical and economic evaluation, the adoption of these development factors in the design stages could certify a first tier of quality to at least guide individuals in their choices. This approach has already been fostered by health care authorities in their initial attempts to support individual app choices, as in the case of the NHS App Library and DiGA Directory. However, trials engaging MMAs for chronic disease self‐management were found to seldom adopt developmental factors or report on them (Adu et al., 2018; Cucciniello et al., 2021).

Contrary to this, we recommend that considerations related to the design and development of MMAs should not only be acknowledged by developers but also be part of the multidimensional assessment upon market entry. Shared decision‐making approaches to the development of apps can enhance the replicability of apps and ultimately lead to improved outcomes (Adu et al., 2018; Edbrooke‐Childs et al., 2019; Joiner & Lusch, 2016). While the adoption of these factors should characterize the lifecycle assessment of MMAs, decision‐makers and scholars need to ensure that the additional value generated thanks to the continuous involvement of users and professionals in the development stage is confirmed empirically and adequately promoted (Table 4).

Recommendation 2. Classification and evidentiary requirements: An unambiguous classification of MMAs should be associated with corresponding evidence generation requirements and the possibility to flexibly review the associated level of risk of every single MMA

The development stage should be followed by the identification of the MMA class of risk and the associated requirements to be considered along the product lifecycle. Among these, clinical validation constitutes the core of thorough assessment processes of any health care technologies, including MMAs. A large debate has focused on whether exceptionalism should be applied to the clinical and economic evaluation of MMAs. Is DH different and even at odds with methodological rigor, or should the same rules in place for other healthcare technologies apply? (Greaves et al., 2018). The assessment modules for MMAs should include additional technology‐specific items is undisputed (Moshi et al., 2020). What is arguably more controversial are the evidence requirements that MMAs should adhere to and, above all, when this should happen along their product lifecycle. This issue is under considerable debate, given the variety of available apps that differ, among others, according to the main functionalities adopted, behavior change techniques implemented, involvement of health care professionals, and level of technology automation in decision‐making, all of which contribute to define the expected level of risk. As for the evidence requirements, MMAs should comply with the same evidential requirements in place for the equivalent medical device class, depending on their risk level.

In this sense, the most mature example of an all‐encompassing classification is the NICE ESF, which aims to help innovators and commissioners by delineating the corresponding levels of evidence (NICE, 2018). However, it was demonstrated that the specificities currently reported in the standards may not be sufficiently suitable to unambiguously group mobile apps according to their risk tier (Nwe et al., 2020), and at least one review noted that peer‐reviewed publications assessing MMAs found that few meet the required evidence level set out by the NICE framework (Forsyth et al., 2021), reflecting the fact that major barriers still need to be overcome for the solid translation of the NICE, or any other, methodology into practice.

Recommendation 3. Performance and maintenance monitoring: MMAs must assure that technical system implementation does not threaten overall effectiveness. Analytical validity must be determined by the manufacturers and a core set of indicators should be developed coherent with the MMA classification

Technical soundness is highly significant for mobile technologies for multifold reasons: (i) limited constraints in terms of manufacturing and distribution, together with the high scalability of MMAs, make full‐scale adoption of apps possible and easier than it has ever been for previous technologies; (ii) most of the app solutions are developed to be managed by patients directly and require additional control mechanisms compared to traditional ones; and (iii) apps operate in complex environments, in which frequent changes and modifications can be implemented, often based on the output generated through the device. Process monitoring and performance testing thus refer to the “continuous process of collecting and analyzing data to compare how an intervention is being implemented compared to its expected results” (WHO, 2016). The technical robustness of any MMA is fundamental to guarantee the delivery of intended results in an accurate and reliable way. Although this concept has been stressed by all reviewed documents, at the empirical level, experiments have shown that MMAs regularly fail to perform as intended, often resulting in failure or crash (Hussain et al., 2018). More recently, an analysis performed on Android contact tracing apps deployed by European countries demonstrated that these apps were not free of weaknesses, vulnerabilities, and technical misconfigurations (Kouliaridis et al., 2021). As of today, definitive standards to guide developers and regulators in their respective duties on this specific domain are lacking (Llorens‐Vernet & Miró, 2020), as generic standards available for medical devices are thought to inhibit rather than foster innovation (Van Velthoven et al., 2018).

Given this gap, our recommendation for MMAs is to assure that technical system implementation does not threaten overall effectiveness. This means that analytical validity should be determined by the manufacturer in the verification phase of the software and would need to cover the four major components of monitoring identified by the WHO (WHO, 2016) (Table 3), with suitable metrics that regulatory bodies should check and validate. The collection of robust metrics should progressively help agencies identify a minimum set of standards that all apps should comply with, based on their classification. While performance testing must be a preliminary requirement ahead of market launch, the distinguishing features of MMAs mandate that it be longitudinally monitored, given that app features may evolve, even significantly, over time.

Recommendation 4. Usability should be continuously monitored, both in the development phase of the solution and after its implementation in the field

Usability refers to the quality of the interaction between the user and the technology. MMAs are interactive technologies, whose value is inherently embedded in the direct relationship with end‐users, which can be highly diversified across individuals. Therefore, the actual longitudinal use of the app is itself a prerequisite for the intervention to deliver its promised effects. In fact, continuity in use is extremely challenging for eHealth technologies, to the point that the term “law of attrition” was coined to denote the substantial proportion of users that drop out before study completion (Eysenbach, 2005). MMAs are no exception, as only small portions of users show consistent interaction over time. A meta‐analysis on drop‐out rates in app‐based interventions for chronic diseases showed a pooled drop‐out rate of 43% (95% CI 29‐57), emphasizing that this issue is inadequately investigated, with a likely under‐estimation of the true extent of the phenomenon, higher observed dropout rates in real world settings compared to RCTs, and limited examination of the reasons behind attrition (Meyerowitz‐Katz et al., 2020). Similarly, in the context of mental health, apps showed even poorer user retention, with a median 15‐day retention of 3.9% and 30‐day retention of 3.3% (Baumel et al., 2019). A further challenge regards the optimal level of interaction, which is necessarily subjective (Michie et al., 2017), as the same levels of interaction may generate different outcomes across categories of individuals.

To sustain the ability of MMAs to produce intended benefits, we recommend that usability should be monitored both during the development stage and after the field implementation of the technology. Usability assessment should hence epitomize the advocated lifecycle approach. Before placing the product on the market, MMA usability scales should be collected to evaluate self‐reported usability by the intended end‐users and professionals. Numerous scales have been proposed in the literature (Azad‐Khaneghah et al., 2020), some of which have been designed and validated to specifically assess the usability of MMAs (Zhou et al., 2019). The monitoring of usability should, however, continue for the entire lifecycle of the MMA, with scales collected over time to monitor effects due to modifications in the technology and its content, coupled with appropriate usage metrics (e.g., access, number of logins, attrition levels), as indirect measures of real‐world usability.

Recommendation 5. Additional outcome domains: Besides conventional outcomes common for all health care technologies, patient empowerment associated with use and the potential environmental impact are distinguishing outcome domains of MMAs that, where relevant, should be appropriately measured and valued

In terms of outcome domains, relevant outcomes are comparable to those in use for other health care technologies, and standard clinical quality measures should be appraised (Mathews et al., 2019). In accordance, published peer‐reviewed studies mostly target primary outcomes pertaining to the clinical domains and to life impact (Cucciniello et al., 2021). However, the promise of MMAs to go beyond the traditional patient‐physician relationship, in which the latter makes decisions in the patient's best interest, creates the opportunity to gauge additional outcomes during the assessment. First, apps that are directly used by patients and their caregivers promise to significantly impact their empowerment, defined as the process of gaining better knowledge about one's health and being able and motivated to influence it (Tomes, 2007). Patient autonomy (i.e., patients' autonomous health behavior) is recognized by the DiGA Guide (BfArM, 2020b) as a relevant patients' health outcome such that DH can effectively support their involvement in the decision‐making processes. Although several international calls to action have fostered the adoption of DH to enhance patient empowerment, there is a lack of clarity and consensus surrounding measurement of the concept, with no comprehensive metrics yet identified and validated (Morley & Floridi, 2019). Contingent valuation (CV) can represent a viable solution in this case. Although the literature is still rather scarce (Callan & O’Shea, 2015; Fawsitt et al., 2017; Shariful Islam et al., 2016), a recent paper (Somers et al., 2019) has demonstrated that CV is effective in providing decision‐makers with a broad base of evidence concerning what people value in MMAs.

On a similar note, growing attention is being paid to the environmental sustainability impact of mobile apps, although this still needs to be formally incorporated in available guidelines even when they comprehensively map positive care effects (BfArM, 2020a). While the impact of apps may go as far as generating significant reductions in CO2 emissions as a result of less face‐to‐face visits and reduced traveling by both patients and healthcare professionals, increased use of digital technologies is associated with sharp surges in global energy consumption (Chevance et al., 2020). The net impact is nevertheless unknown, as no empirical studies on apps have tried to assess it to date. Digital technologies and MMAs can have a significant impact on health care systems' quest for a sustainable future (The Lancet Digital Health, 2021), but it is time to raise global awareness of their potential and start measuring them.

Recommendation 6. Clinical evidence requirements: Flexible study designs that account for the specific characteristics of MMAs and can generate fast and efficient results should be adopted and coupled with flexible policy arrangements, based on the level of risk and the position in the product lifecycle of the MMA

MMAs that offer unvalidated diagnostic and therapeutic functions with no basis in evidence are in constant observance, as confirmed by the literature (Iribarren et al., 2021; Whitehead & Seaton, 2016). The reasons behind this problem are multiple and largely associated with challenges that make conventional methodologies unsuitable for evaluation (Tarricone et al., 2021).

Regarding RCTs, and prospective studies in general, apps are iterative solutions and hence generally incompatible with locked‐down interventions (Tarricone et al., 2021). Costs and length of randomized studies are considered to be too high, given the perceived level of risk associated with most apps and available resources of developers. Retrospective studies may be easier but have rarely been conducted due to challenges in data access and limited use of MMAs in clinical practice (Guo et al., 2020).

Flexible prospective study methodologies that account for the specific characteristics of MMAs and that can generate faster and more efficient results should be adopted, by both tailoring randomized designs to the requirements of MMAs and leveraging on the opportunities provided by system simulations (Guo et al., 2020). These methodologies include MOST, SMART, micro randomized trials, and more in general trials developed with an adaptive design (Klasnja et al., 2015; Pham et al., 2016). At the same time, MMAs may constitute an ideal setting in support of increasing interest in the use of real‐world data to support regulatory decision making. While the use of real‐world data to support regulatory decision‐making and assessment processes would not be new (long employed to support safety reports and risk management), there is lower acceptability of real‐world data when the outcome of interest is effectiveness (Cave et al., 2019). Greater caution is typically exercised for the evaluation on patient outcomes, and this has typically precluded the adoption of observational data in regulatory processes. Practical and ethical reasons, however, are pushing to shift the regulatory landscape and decision‐making toward greater real‐world data use.

While innovative study methodologies may speed up the evidence generation process and contribute to close the current void, additional questions are related to when evidence should be delivered to maximize MMA uptake in everyday clinical practice.

We believe that flexible arrangements such as that set up by the German Digital Health Act (BfArM, 2020a) and the Pre‐Cert Program (FDA, 2020d) are desirable and should be adopted by more jurisdictions. Early‐stage, low‐cost evidence generation approaches, albeit weaker, should be enough to secure market entry and receive temporary reimbursement. It is then necessary to address two additional topics: a specific deadline by which more robust evidence must be provided to confirm the initial clearance and a monitoring process to accompany the product over its entire lifecycle. Given that MMAs may be continuously revised and possibly change functional classification and risk level, it is necessary to dynamically determine whether additional evidence is needed as recommended for other types of medical devices (Tarricone et al., 2020). An ad hoc agency, directly managed by government bodies or subcontracted to independent third parties, could be charged with monitoring the evolution of commissioned MMAs, authorizing software changes and requiring additional evidence with predetermined timeframes when necessary. This complex governance can only be accomplished as part of collective effort of all parties involved.

Recommendation 7. Safety and security: Risks related to data privacy, cyber security and misinformation and their potential impact must be thoroughly assessed for MMAs

As for other domains, safety acquires a slightly different meaning when applied to MMAs, as apps may generate unwanted effects through multiple channels. Other than the direct harm that could be caused by contact with the device and the undesirable effects that may be generated by diagnostic inaccuracy in medical software, an additional source of risks should be thoroughly assessed. A scoping review by Akbar et al. (2020) filled this gap by highlighting that available apps do indeed pose clinical risks to consumers with potential consequences on patient health. The analysis revealed a total of 80 safety concerns, the vast majority of which related to the quality of the content present and the appropriateness of information sources included within the apps. The dissemination of low‐quality information, through either incorrect, incomplete, or inconsistent data, may be harmful to consumers by promoting inadequate or inappropriate health behaviors (BinDhim & Trevena, 2015). Since lay persons may not be able to distinguish whether the included information is correct or misleading, some of their decisions may be ill‐managed and may induce them to seek help with considerable delay or inappropriately orient individual choices.

Pending further discussions and elaborations, methods to certify the accuracy of conveyed content and, thereby, the safety of apps have not yet been adopted by regulatory agencies and decision‐makers. While MMA safety should be pre‐certified before market entry, a vigilance framework that collects safety concerns for MMAs along the entire lifecycle should complement the monitoring.

Despite being strictly intertwined, the direct and indirect impacts of MMAs on patient safety through inadequate content delivery should be kept separate from security concerns, which are indeed significant for MMAs as for any digital technology. Although current assessment processes (e.g., BfArM, 2020b; NHSX, 2021) do prescribe strict adherence to data protection regulation systems (e.g., General Data Protection Regulation [GDPR]), several studies have reported that commercially available apps shared personal data with third parties, often for economic gain, and could compromise the privacy of users (Grundy et al., 2019; Huckvale et al., 2015).

Recommendation 8. Equity considerations: given the context of escalating inequalities and persistent technological divide, it is paramount to evaluate the net effect of MMAs on equity

Great expectations have emerged for MMAs, and DH in general, for their potential to enhance access to care and reduce inequities. Improvements in health care technologies often tend to cause disparities, as was certified by the COVID‐19 pandemic (Watts, 2020). Without empirical evidence, it is nonetheless hard to predict whether apps will support progress toward equity or instead exacerbate existing inequities (Makri, 2019). On April 17, 2019, the WHO (2019) released the report “WHO Guideline: recommendations on digital interventions for health system strengthening,” to guide the evaluation process of evidence on emerging DH interventions that contribute to health system improvements, based on an assessment of benefits, harm, acceptability, feasibility, resource use, and equity considerations. The recommendations examine the extent to which DH interventions can expand universal health coverage and focus on a subset of prioritized DH interventions accessible via mobile devices that, however, do not pertain to MMAs (i.e., birth notification via mobile devices, death notification via mobile devices, client‐to‐provider telemedicine, provider‐to‐provider telemedicine, targeted client communication via mobile devices, digital tracking of patient/client health status and services via mobile devices, health worker decision support via mobile device, provision of training and educational content to health workers via mobile devices, and mobile learning–mLearning). More recently, the WHO (2021) has reiterated the relevance of equity of access and inclusion in the “Ethics and Governance of Artificial Intelligence for Health.” The UK NICE (2018) refers to equity when it recommends including subgroup analyses to show the relevant economic impact provided the availability of good clinical data to show that the effects differ by demographic factors.

The equity domain should hence examine the impact of the app in various clinically and socially relevant subpopulations. The impact may vary across different components in the assessment: the design component pertains to the possibility to personalize some of the technical features of the app (such as font, size, and colors), to facilitate the accessibility to disadvantaged groups; the technical component is related to both data consumption from app utilization and network characteristics, coverage, and data transfer speed in a particular area; the strategic component, which may allow for pre‐assessing the potential equity impact of the app and adopting timely informational and educational strategies to mitigate its risks; the cultural component, which underlines the inherent disadvantages of specific social groups and aims at making the benefits of DH more accessible.

Recommendation 9. Organizational assessment: The assessment of direct and indirect implications of the adoption of MMAs on the organizational level should focus on process, people, structure, culture or management impacts

With the exception of the UK (NICE, 2018) and Germany (BfArM, 2020b), the majority of the currently available frameworks for the assessment of DH neglect the organizational implications associated with their adoption. On the contrary, greater emphasis should be placed on this domain by both decision‐makers and producers. Attention to organizational impact should cover the entirety of app‐based interventions, including those providing telemedicine services, whose adoption has been significantly accelerated by large‐scale accessibility of smartphones at the patient level (Allaert et al., 2020). The expected extent of organizational transformation induced by MMAs is extremely diversified, with MMAs variously impacting the way treatments are organized, the responsibilities of different health care professionals, and the amount and combinations of to employ. It is not possible to simply drop MMAs in established organizations without any tailored adjustments. Two interconnected points of interest pertain to the impact of apps on integrated care (IC) and their interoperability with existing systems. MMAs can be the cornerstone for allowing IC, thus reshaping chronic care and securing stronger data‐based connections between levels of care. Initial acceptability studies of mHealth‐enabled IC programs that involved all health care professionals from different levels and organizations have shown that, while patient‐reported acceptability was extremely high, professionals criticized the lack of maturity and integration with legacy systems (de Batlle et al., 2020). Advances in mHealth are instead dependent on interoperability (Lehne et al., 2019), which should be a prerequisite for the reimbursement of any app and part of their assessment processes.

Recommendation 10. Cost and economic evaluation: The evaluation of the cost and economic impacts of MMAs should require no innovative forms and methodologies but rather new metrics for outcomes and different structures for cost

Formal economic evaluations of the value for money associated with the introduction and uptake of health technologies are a cornerstone of the HTA paradigm (e.g., EUNeHTA HTA Core Model®) but were not explicitly cited by frameworks for mobile apps mapped by Moshi et al. (2018) and were not considered in the policy documents reviewed, with the exception of the UK ESF. In a more recent systematic review, Vis et al. (2020) found that several frameworks do include the economic dimension. However, the review focused on eHealth technologies and most of those frameworks covered telemedicine programs (Vis et al., 2020). No ad hoc metrics have been proposed for the economic evaluation of mHealth apps, nor should they be, in our opinion. Since MMAs compete for the same, restricted bulk of resources as all other health care technologies, the methodologies, and forms adopted should be similar and make comparisons possible across different technologies. What should change are the types of costs to be accrued, as well as how the suitable comparator is chosen. As for the former, the cost structure for MMAs is reportedly different from that of other health care technologies due to their potential to change scale at reduced incremental cost and the need to account for specific cost items, such as those associated with design and promotion for the active and continuous uptake of the app (Tarricone et al., 2021). Specific challenges are also associated with the selection of the comparator. No one‐size‐fits‐all solutions are practicable. Standard‐of‐care technologies may sometimes be the best comparator for apps that have no major impact on the entire process of care, such as those meant to improve the level of awareness regarding a certain condition or activate individual self‐management. For those that are significantly integrated with care processes, instead, the identification of the best comparator might be extremely intricate: these apps provide complex interventions that may alternatively aim at simply enhancing existing pathways with additional components, refashioning them or even entirely substituting the way they have traditionally been offered. Depending on which is the case, costs and outcomes to include in the cost‐effectiveness analysis will have to change accordingly.

Preliminary analyses from the scientific literature confirm that the evidence to support cost‐effectiveness is still insufficient and that estimating program costs and outcomes is extremely challenging (Hazel et al., 2021). Technological progress typically contributes significantly to the rise in health care spending: DH and apps could overhaul this trend, despite potential increases in demand and use associated with these technologies may complicate what appears to be an elusive quest for cost savings (Rahimi, 2019).

DISCUSSION

Next Generation EU (European Commission, 2021), the €750 billion recovery instrument fostered at the European level in response to the COVID‐19 pandemic, will support health digitization and further boost the development of MMAs. DH and namely MMAs have a strong international and cross‐border element, to the point that they have been considered a “cross‐jurisdictional” practice of medicine (Khirasaria et al., 2020). Their immateriality and compatibility with widely available consumer technologies make them prone to easy—global—scalability. The DH market is estimated to grow by 25% CAGR from 2019 to 2025, experiencing a four‐fold increase from 175 to 657 bln USD (Global X [ETFs], 2021). Hence, despite the need for context‐specific considerations and adjustments, MMAs demand converging international principles and the building of worldwide consensus on a set of recognized standards, ensuring consistency around their understanding, regulation, and assessment.

The targeted review of policy documents revealed that current assessment frameworks present some common elements, especially those that have a regulatory scope, but also significant differences.

As for the target focus of regulatory frameworks, software, and not MMAs (with the exception of Germany), is gaining traction, suggesting the relevance of the work conducted by the IMDRF in centering regulatory formation around SaMD‐derived concepts. The US, for instance, is a clear example of this influence, as proven by its switch of focus from MMAs toward the more encompassing area of DSWF.

Moving to MMA assessment, identified approaches are widely different and IMDRF influence is far less noticeable. Searching for common themes, guidelines span both clinical and technical domains, and strive toward a continuous risk‐monitoring approach through real‐world data. A common challenge regards addressing iterative updates, where the US approach of complementing a firm‐centered approach through quality and organizational assessment and structured real world performance requirements appears to be the only tentative solution. The UK, through its ESF, provides the most comprehensive and detailed guideline for effectiveness and economic assessment, including indications regarding the type of mandated study designs. Nevertheless, the study designs do not differ from those identified for other technologies (e.g., medical devices). We believe instead that broadening the scope of outcome domains—mainly clinical—typically considered in evaluation and contingent valuation could represent a better option for evaluating the benefits of MMAs. The German approach may have possibly taken inspiration from the UK approach, as observed in the detailing of accepted study designs, with the setting of both minimum and preferred requirements. Interesting to note is that the DiGA Guide is the only example emphasizing the relevance of alternative trials, such as PCT, SMART, and MOST, for the evaluation of DH devices. However, while the types of studies are addressed in some guidelines (e.g., NICE, DiGA), study designs are currently less regulated and future assessment frameworks need to offer more guidance (e.g., what are the appropriate population, intervention, control, and outcomes) at this regard so to generate relevant high‐ quality clinical evidence. Economic evaluation is, however, missing from assessment purposes and possibly considered only throughout price negotiations.

At the broader European level, a remarkable innovation in the assessment process of MMAs may come from the new EU regulation on HTA, tentatively approved in 2021 (European Commission, 2018; European Council, 2021). The HTA regulation comes as a follow‐up to over 10‐year of voluntary collaboration among HTA agencies within the Joint Actions EUNETHA context, a project‐based framework for the sharing and harmonization of processes, procedures, and methods. It is likely that the governing body (i.e., the HTA Coordination Group) would continue to adopt the HTA Core Model® to assess health technologies, but it is not clear whether it will be updated to take into consideration the specific features of MMAs compared to all other products (i.e., drugs and medical devices). It would be advisable that the new HTA Coordination Group strictly collaborate with the Medical Device Coordination Group of the EU MDR because the iterative nature of software development makes it unsuitable for a linear, conventional, and assessment approach based up the pre‐market and post‐market assessment phases. Rather, a circular process that covers different steps within the entire product lifecycle is needed to generate evidence in support of market authorization, coverage, reimbursement, and purchasing decisions. Regulation and assessment are part of a greater innovation chain, which, starting from early prototyping to product commercialization, then translates market responses into conducive inputs to ideation. Considering this, as long as jurisdictions lack a proper systematization of the broader DH landscape, be it in form of taxonomy or mapping, the risk is that mainstream health care sector stakeholders may interpret DH as part of vertical, independent, and disconnected strategies (Labrique et al., 2013). Germany and the UK represent pioneering exceptions in this regard, especially in terms of assessment. To promote further uptake following examples such as these, a multi‐dimensional perspective that also involves HTA is necessary to examine the impact of MMAs on all relevant outcomes (Vis et al., 2020).

Among recent contributions, Moshi et al. (2020) developed a dedicated HTA module aimed at making thorough evaluation of MMAs possible, emphasizing the need to adopt a tailored approach with the goal of facilitating their generalized acceptance as part of standard clinical care (Moshi et al., 2020). Similarly, Haverinen et al. (2019) have proposed their Digi‐HTA model, pinpointing the domains needed in this assessment, by combining the HTA core processes with DH care services and highlighting the need for agencies and policymakers to develop standardized, evidence‐based assessment processes (Haverinen et al., 2019).

Like the two previous contributions, our objective is to support decision‐makers, and producers in the introduction of MMAs in clinical care processes. Our recommendations, however, do not specifically aim at adapting current assessment processes and modules to develop fully dedicated ones for MMAs but rather embrace the lifecycle approach that is needed to fruitfully govern them. As such, we reckon that a preliminary requisite should be the definition of a dynamic, unambiguous taxonomy that groups MMAs, depending on prespecified criteria. Ad hoc evidence requirements should then be attached to each of these groups and continuously updated based on technological advancements and MMA updates that result from performance monitoring. It is beyond doubt that MMAs retain a groundbreaking potential for disrupting health care paradigms and nurturing healthier populations, but this materialization should not be taken for granted. Governments and societies at large hold the unique responsibility of making stronger efforts to ensure the sustenance of cost‐effective and efficient digital ecosystems. Our 10 recommendations aim at providing decision‐makers and all stakeholders with key elements, beyond economic considerations alone, that should be included in any assessment framework for MMAs which are meant to contribute to the development of cost‐effective DH. The implementation of the EU MDR in May 2021 together with the EU HTA Regulation represent a precious opportunity for the EU to endorse these principles.

CONCLUSION

The growth of DH technologies, accelerated by the COVID‐19 pandemic, has been (although not proportionally) paralleled by the publications of assessment frameworks by regulatory and HTA bodies as well as by HTA scholars. Nevertheless, most of these frameworks tend to cover the broad range of software‐based technologies (e.g., SaMDs), thus leaving ample room for ambiguity, uncertainty, and indecisiveness to the applicability of requirements to specific products such as MMAs. Medical mobile apps present distinct characteristics that make them different from other MDs and SaMDs (e.g., telemedicine) and need to be considered when their assessment is at stake. A comprehensive framework that covers the full range of the lifecycle of MMAs and all relevant domains, including those less commonly considered, such as environmental impact and equity implications, must be addressed. Our 10 principles can help improve the development of assessment frameworks for MMAs, especially in view of the upcoming HTA Regulation that together with the recent MDR represent an opportunity for the EU to further expand the work done by EUnetHTA.

CONFLICT OF INTEREST

The authors declare they have no conflict of interest.

Supporting Information S1

Click here for additional data file.

Supporting Information S2

Click here for additional data file.