Economic Evaluation of Digital Health Interventions: Methodological Issues and Recommendations for Practice.

Health care interventions are increasingly being delivered through digital technologies, offering major opportunities for delivering more health gains from scarce health care resources. Digital health interventions (DHIs) raise distinct challenges for economic evaluations compared with drugs and medical devices, not least due to their interacting, evolving features. The implications of the distinctive nature of DHIs for the methodological choices underpinning their economic evaluation is not well understood. This paper provides an in-depth discussion of distinct features of DHIs and how they might impact the design, measurement, analysis and reporting of cost-effectiveness analysis conducted alongside both randomised and non-randomised studies. These include aspects related to choice of comparator, costs and benefits assessment, study perspective and type of economic analysis. We argue that typical methodological standpoints, such as taking a health service perspective, focusing on health-related benefits and adopting cost-utility analyses, as typically adopted in the economic evaluation of non-digital technologies (pharmaceutical drugs and medical devices), are unlikely to be appropriate for DHIs. We illustrate how these methodological aspects can be appropriately addressed in an evaluation of a digitally supported, remote rehabilitation programme for patients with Long Covid in England. We highlight several methodological considerations for improving practice and areas where further methodological work is required.

Key Points for Decision Makers

Introduction

The number of health interventions delivered through digital technologies, such as smartphones, web-based resources and text messaging, has increased exponentially in the last few years. Such digital health interventions (DHIs) have been used to facilitate remote access to effective treatments [1-3], improve the management of chronic conditions [4, 5] and promote healthy behaviours [6, 7]. DHIs are often intended to augment or replace traditional, face-to-face health care interventions, and may be paid for out of healthcare budgets, particularly in single payer systems such as the UK National Health Service (NHS). In these settings, economic evaluation studies are of central importance to assess the extent to which DHIs provide good value for money compared with alternative options.

Most published economic evaluations of DHIs adopt standard methodological recommendations for the evaluation of health care technologies, such as pharmaceutical drugs and medical devices [8-10]. This often includes taking a health service or payer perspective [11], focusing on health-related patient benefits [12] and adopting cost-utility analyses [13]. However, these methodological assumptions may not reflect the distinct nature of DHIs. For example, DHIs are typically complex interventions, composed of multiple interacting components. Hence, establishing their cost effectiveness will require a much broader assessment of costs and effects, including non-health benefits and costs falling outside the health care sector [12, 14]. In addition, the costs and benefits of DHIs are often a function of how the intervention evolves over time and the extent to which the user interacts with them, so impacts are likely to be more heterogeneous [15, 16].

There is a lack of guiding principles on how to appropriately conduct economic evaluations of DHIs. LeFevre and colleagues [17] have proposed a simplified algorithm to guide analysts through key stages of the economic evaluation of mobile applications, such as whether a full or partial economic evaluation should be undertaken. McNamee and others [9] have discussed the implications of applying a complex intervention framework to the economic evaluation of DHIs. Both papers focused on aspects related to decision analytical modelling of DHIs. The implications of DHIs for the methodological choices underpinning economic evaluations using individual patient data from randomised controlled trials (RCTs) and observational studies remains poorly understood.

The aim of this study is to provide a more in-depth discussion of several methodological aspects of economic evaluations of DHIs based on individual patient data analyses. The focus of the discussion is on health care systems where DHIs explicitly compete with non-digital interventions for health care resources. The objectives of this study are (i) to review the distinct aspects of DHIs compared with common healthcare technologies, such as pharmaceuticals and medical devices, (ii) to discuss the implications of that distinct nature to the methodological choices underpinning economic evaluations of DHIs, (iii) to illustrate how these methodological points can be incorporated into an economic evaluation of DHIs and (iv) to highlight key methodological considerations for improving practice and identify areas where further work may be required.

Distinct Aspects of Digital Health Interventions (DHIs) and Implications for Economic Evaluation

This section discusses key distinct features of DHIs compared with pharmaceutical drugs and medical devices, as these have been of prime interest to HTA agencies. We define DHIs as interventions that aim to improve the health of users and are delivered through digital technologies such as computer programs, mobile phones applications and websites. The simplest forms of DHI focus on supporting diagnostics and remote disease monitoring [18]. Many others involve digital tools, such as mobile apps and web resources, to support individual behaviour change (e.g. smoking cessation and reducing alcohol use) and the management of long-term conditions [4-7]. More complex forms of DHIs provide remote access to widely used therapies, such as cognitive behavioural therapy and physiotherapy [1, 2, 5]. This paper focuses on the latter types of DHIs that aim to treat, promote or manage health through supporting behaviour change and decision making of the patient or health care professionals. Such interventions are typically interactive, personalised, based on user input and may be provided at the individual or population level [19]. Simpler forms of DHIs, such as diagnostic and monitoring devices, are much closer to the nature of conventional medical devices, and conceptual issues in the economic evaluation of these have been discussed elsewhere [20, 21].

The key differences between DHIs and non-digital health interventions, such as pharmaceuticals and medical devices, and their implications for economic evaluations are summarised in Table 1 and discussed in greater detail below. The discussion of each of the methodological issues is based on the author’s experience with designing and analysing economic evaluation of DHIs [22-27] and development of guiding principles for the evaluation of DHIs [9, 15, 28], and several systematic reviews of the applied literature across different health settings [8, 11–14, 29, 30]. Illustrative examples for each of these points are summarised in Table 2. These case studies are not meant to be a representative sample of economic evaluations submitted to HTA agencies.

Table 1

Key differences between pharmaceuticals or medical devices and digital health interventions (DHIs) and implications for their economic evaluation

	Pharmaceuticals	Medical devices	DHIs	Implications to economic evaluation
Comparator	Usually a well-defined comparator, e.g. placebo	Usually a well-defined comparator, e.g. competing device	Often a combination of alternative treatment options	To consider both digital and non-digital comparators and whether DHI replaces or complements existing technology
Product evolvement	Fixed	Evolves gradually with product modification, and innovation	Evolves fast with user feedback and requires frequent updates	To account for the rapid evolution of DHI and its impacts on costs and benefits, and the timing of the analysis
User involvement	Generally limited to compliance	Interaction between user (e.g. surgeon) and device may or may not be required	Active user input (patient or doctor) always required for DHI to be used as intended	To consider user time (costs) and user experience (benefits)
Intervention cost	Fixed unit price, reflecting both fixed and variable costs	Fixed unit price, but dynamic pricing due to weaker regulation than for pharmaceuticals	DHI is often provided at scale. The unit price is the marginal cost, which tends to zero	Development costs not always included in cost analysis. Mean cost per user should be based on the eligible population and expected uptake rates
Benefit assessment	Most benefits reflected by individual health changes	Non-health benefits limited to some products, such as diagnostic devices	DHIs typically lead to diffused health and non-health changes	To include non-health benefits, both to patients and other parties (e.g. health professionals, carers)
Non-health care impacts	Often low; limited to some disease settings, e.g. mental health	Often low, limited to some interventions, e.g. cardiac devices	Often significant, such as productivity impacts, irrespective of the setting	To consider all relevant impacts outside the health care sector as part of an ‘impact inventory’
Economic analysis	Cost per QALY assessments usually appropriate	Same as pharmaceuticals	Cost per QALY unlikely to reflect broad range of health and non-health impacts	Cost-consequence analysis is likely to be most suitable and in line with an impact inventory

QALY quality-adjusted life years

Table 2

Case studies illustrating how the distinctive features of the DHI were incorporated into the economic evaluation

Methodological issue	Digital health intervention	How the distinctive feature of the DHI was incorporated into the economic evaluation
Comparator	The HeLP-Diabetes programme [23]: internet-based intervention to support self-management of type 2 diabetes in England. The intervention included an educational, interactive website, with behaviour change support, emotional support and training sessions to facilitate access to the HeLP-Diabetes tool	The intervention was designed as an addition to current practice, which might have included general information provided by the GP or available online. The comparator group was defined as publicly available online information based on both Diabetes UK and NHS Choices websites. To help comparability between the intervention and ‘usual practice’, participants in the control group were also given an introductory facilitation meeting to help them navigate these websites and an information booklet to take home
Product evolvement	Down Your Drink [27]: internet-based intervention to support behaviour change in people with hazardous alcohol consumption. This consisted of an online interactive programme (weekly interaction to read materials and complete exercises) based on cognitive behavioural techniques	Following user feedback, major components associated with the development of new modules and features to improve the attractiveness and functioning of the website had to be introduced. This included re-structuring website components, adding new features to improve user interaction and implementing a new incentives system. As a result, the additional costs involved with user-led redevelopment led to the intervention cost being twice as big as that initially estimated [45]
User involvement	Digital smoke cessation programme [46]: web-based intervention to support smoking cessation, with or without counselling. The economic evaluation included cost impacts to the patient arising from active user involvement	Data were obtained by measuring and costing (i) the resources associated with travel to smoking cessation sessions and time spent engaging with the smoking cessation digital platform, (ii) informal care, which accounted for potential additional support by the caregiver to interact with the website. On the outcomes side, any health impacts resulting from user involvement were assumed to be captured in the patient-reported quality of life outcomes
Intervention cost	The Link tool [47]: web-based mental health navigation tool (Link) to guide young adults with severe mental distress to appropriate online and off-line sources of mental health information and care. The web platform has been developed exclusively for this intervention	Development costs were included and were very high (AU$1.74 million) compared to the maintenance costs (AU$29,803). The marginal cost of providing Link was essentially zero, and hence the study used an estimate of the population likely to receive the intervention and uptake rates to estimate the mean cost per user. This led to a low cost per user (around AU$5)
Benefit assessment	The ESTEEM programme [48]: online triage system, led by either a GP or nurse, for managing same-day consultations in primary care. The economic evaluation considered both health and non-health benefits of new digital patient management system	Non-health benefits included aspects related to system efficiency (e.g. health care contacts required to treat patient), user experience (e.g. care readiness), wellbeing (convenience of care) and problem resolution. While differences in patients’ health status (EQ-5D-3L) were small between consultation systems, GP and nurse-led online triage led to much higher patient satisfaction and problem resolution scores
Non-health care impacts	Web-based perioperative recovery [49]: Bouwsma and colleagues developed and assessed a web-based care programme to facilitate recovery of women undergoing gynaecological surgery. This digital intervention was anticipated to have a significant impact on women’s ability to return to work	Study adopted societal perspective and included a broad range of non-health care costs and benefits associated with swifter return to work. This involved quantifying the time to ‘sustainable return to work’ and any costs savings associated with both absenteeism and presenteeism. Absenteeism and presenteeism costs were calculated using the human capital approach (equivalent to sick leave costs). The productivity-related costs (£8443) represented about 70% of the total cost of the intervention (£12,266) and drove the cost savings (− £647) of the DHI compared with usual care
Economic analysis	The ESTEEM programme [48]: online triage system, led by either a GP or nurse, for managing same-day consultations in primary care. This study nicely illustrates how CCA can help inform decision making when non-health benefits are of prime interest to the target population	CCA allowed decision makers to assess relative value for money of the new digital patient triage system according to the benefits they wish to prioritise. For example, if the priority was to benefit overall GP workload, then GP or nurse-led online triage was unlikely to be cost effective compared with standard telephone triage, because it just changed the nature of that workload. Conversely, if the decision maker was more interested in benefits in terms of reducing GP visits, nurse-led online triage was likely to provide good value for money

CCA cost-consequences analysis

Choice of Comparator

While the challenges associated with choice of a comparator are not exclusive to the economic evaluation of DHIs, there are some distinct aspects to consider when defining a comparator in this context [8, 30]. Irrespective of the purpose of the DHI, the choice of comparator will be a function of how the intervention interacts with non-digital health care. For example, the DHI might complement or substitute face-to-face health care delivery and paper-based information systems. In settings where the intervention is implemented in a space already dominated by digital care, a relevant comparator might be (i) an alternative way of implementing the same DHI, (ii) a competing DHI or (iii) an existing technology that the DHI is replacing. DHIs differ from other non-drug interventions (e.g. medical devices) in terms of the difficulty in isolating the specific intervention from the range of potential digital inputs.

A more pragmatic aspect related to the evaluation of DHIs and choice of comparators is the source of costs and effects and study design. RCTs may not often be feasible to evaluate DHIs as these tend to rapidly evolve as they are implemented. As a result, comparisons between DHIs and alternative options will often come from non-randomised studies, which raises additional methodological (statistical) concerns, such as selection and confounding issues. In this context, quasi-experimental designs, such as difference in differences (see example [31]), should be preferred as they can help tackle both measured and unmeasured confounding. Alternative study designs and evaluation methods of DHIs are discussed elsewhere [32].

Product Evolvement

In contrast with pharmaceuticals, DHIs evolve significantly due to user feedback and technical enhancements. If a digital product remains unchanged for some time it risks becoming out of date or inoperable, and is likely to lose some of its usefulness as well as economic value. Updates may focus on (i) use and uptake aspects, such as look and feel, navigation and rewards for use, (ii) software components, to maintain compatibility with operative systems or web browsers or (iii) active content features, for example changing advice as research evidence evolves. Evolvement over time is not exclusive to DHIs [21]; for example, medical devices can also undergo product modifications, but DHIs can be seen as on the extreme end of the spectrum in terms of the frequency and extent of product evolvement.

Therefore, the economic evaluation will need to take into account the change in key product components and direct implications for resource use and benefit measurement and assessment [14]. For example, this can be incorporated at the design stage by carefully devising a health economic analysis plan and developing flexible data collection tools that can accommodate new, sudden impacts to costs and effects as a result of product evolvement. Another important consideration is the timeframe in which the expected impacts are likely to occur. The economic evaluation should consider both the relevant healthcare pathway (as per non-digital interventions) and the life cycle of the DHI. The timing of the analysis will often involve choosing a period when the DHI is likely to have reached a ‘steady state’.

User Involvement

DHIs are likely to be more effective when they are accompanied by human support to ensure the digital product is used as intended [16]. However, human input may not necessarily add value because unguided interventions may still be effective (although the magnitude of the effect is likely to be smaller) and be provided at a lower cost at the population level. On the one hand, human support has immediate implications to the variable costs of providing DHIs. This can include the additional resources incurred by the health system to support use of DHIs (facilitation costs), or the time spent by the user as a form of productivity loss (opportunity cost of using DHI) [12]. These costs can be considerable, depending on the extent and frequency of involvement required, and directly affect the size of the marginal cost of providing DHIs.

On the other hand, user involvement may be an important component of the ‘effectiveness’ of the DHI [15, 16]. The benefits can be affective, for example by creating an intrinsically enjoyable user experience (e.g. health promotion), or functional by improving the perceived benefits of the digital intervention itself (e.g. cognitive behavioural therapy). Irrespective of whether user involvement should be seen as a cost or a benefit (or both), this distinct aspect of DHIs strengthens the case for adopting a wider perspective in their economic evaluation.

Intervention Cost

Most health care technologies face falling average costs as scale increases, reflecting a much larger proportion of fixed costs compared with the variable costs. For example, manufacturers of drugs and medical devices invest up front in research and development, with some compounds failing to progress to market authorisation. The greater the share of fixed costs, the more the average cost will fall with scale. The provision of digital products may also be associated with a high proportion of fixed costs (e.g. development of digital platforms) compared with that of variable costs (e.g. maintenance), but there are several aspects that are distinct to the economic analysis of DHIs.

Firstly, the amount of fixed costs will depend on whether the DHI requires the development of a brand new product or a modification of an existing one. If the digital product is developed from scratch then an appropriate measurement and costing of resources incurred with research and development are required. In this case, the price (unit cost) set by the digital firm may take those into account, similar to that for drug or medical devices. A key distinction is that the digital company may decide to bear the fixed costs in return for health commissioners agreeing to barter user data in exchange, for example for commercial advertising. On the other hand, if the DHI is a modification of an existing product, fixed costs related to research and development are usually excluded, that is, ‘sunk costs’.

Secondly, DHIs operate in a market in which the marginal cost (extra cost per unit produced) of providing the digital product for each additional user tends to zero. For example, the change in variable costs of a mobile app, such as those related to maintaining and updating the app, will approximate zero as the number of app users increases. Therefore, digital health products are often provided at very low cost or free of charge, where the marginal cost is in effect the unit price. This is unlikely to be the case for drugs or medical devices because the price is often set above marginal cost, for example due to monopolistic power, or because the change in costs per additional units produced is unlikely to approximate zero. To enjoy the advantages of low marginal costs, digital companies will want the digital interventions to be used at scale.

Thirdly, for resource allocation purposes, it is often of prime interest to calculate mean differences in total cost between a DHI and alternative options. Given the potentially high fixed costs and low cost of scaling up the digital intervention (marginal cost near zero), the mean cost can be obtained by using the participants in the study or the population that the digital product is likely to reach. The concern here is the potential overestimation of the mean cost per user, particularly when the number of participants in the study is small.

Fourthly, DHIs may not require the same level of investment in infrastructure and training compared with that of medical devices, but they may have a considerable logistical effect on the way health care is delivered. In principle, DHIs are anticipated to lead to efficiency gains in the health system, for example by improving productivity through (i) more output (care) for the same level of inputs (e.g. infrastructure and staff) or (ii) same output using less input, say less staff time). However, they can also lead to considerable additional costs, for example, a new system to remotely deliver GP consultations may require large investment in a new digital patient management system.

Benefits Assessment

Economic evaluations of pharmaceuticals and medical devices tend to focus on the benefits to patients’ health. The effects of DHIs are, however, likely to be much broader, including impacts that go beyond health and the targeted patients themselves [11–13, 29]. For example, DHIs are increasingly being considered to improve health care delivery, such as sharing relevant health information with the patient quicker, and remotely monitoring chronic diseases. Hence, many DHIs are likely to lead to significant benefits beyond individual patients’ health. This may include efficiency gains in the health system (e.g. GPs able to manage more patients with existing resources), or improvements in the quality of care provided (e.g. help patients feel more empowered and capable of managing their own condition).

In addition, DHIs may indirectly impact other patients not initially targeted by the intervention. For example, an internet-based intervention for weight loss for patients with type 2 diabetes mellitus may allow users to share their experiences with other patients with different chronic conditions wishing to reduce weight. DHIs can also have an impact on other parties, such as carers, for example by increasing the amount of informal care required to support interaction with DHI. These unintended effects (also known as spill-over effects) are generally small for pharmaceuticals because they typically affect only the treatment recipient, but can be considerable for DHIs.

Non-Health Care Impacts

By facilitating remote access to treatments and supporting health care delivery, important costs and benefits of DHIs are likely to fall outside the health care sector. For example, mobile apps targeting alcohol use disorders can have significant non-health care impacts, such as productivity gains, greater social inclusion and crime reduction. Even simpler DHIs, such as a mobile app to monitor hypertension, may have important impacts on productivity, out-of-pocket costs and patient empowerment. This emphasises the importance of considering a study perspective that goes beyond that of the health service. This may involve taking a societal perspective [33, 34] that considers the full range of relevant impacts to patients and third parties (e.g. informal carers) across different sectors, such as health, education, criminal justice and informal care. While a wider perspective may also be required in the evaluation of pharmaceuticals and medical devices, this tends to be limited to certain diseases such as mental health. Overall, the complex, interactive nature of DHIs provides a stronger rationale for considering a wider perspective in the evaluation of DHIs compared with that of pharmaceuticals or medical devices.

To make the assessment of non-health care impacts more explicit and transparent, the inclusion of an ‘impact inventory’ [33, 34] should be considered. The main purpose of an impact inventory is to comprehensively catalogue all relevant health and non-health costs and benefits of an intervention, both within and outside the health care sector. The inclusion of an impact inventory would encourage economic evaluations to critically examine the types of relevant impacts and sectors for the evaluation of the DHI.

Economic Analysis

Cost-utility analyses typically combine health effects in terms of both quantity (life years) and health-related quality of life into a single measure, quality-adjusted life years (QALYs). This type of economic evaluation is particularly suitable to inform resource allocation decisions (at a national level) across different health care settings. For example, NICE’s evidence standards framework for DHIs currently recommends using a cost-utility analysis (assuming the health service perspective) as a reference case, when the DHI involves high financial commitment [35].

However, cost per QALY assessment may not be the most suitable approach for the evaluation of DHIs. For example, the effects of many DHIs are diffused and include important non-health benefits that are unlikely to be captured by QALYs. In addition, many DHIs represent a relatively low financial commitment, or the decision to commission a DHI may be at a local level. Other DHIs may represent a low risk to the payer, for example because they are anticipated to be cost saving, in which case cost-utility analysis may not be required. For all these types of DHIs, NICE guidelines recommend the use of cost-consequences analysis (CCA) [35].

CCA typically considers a broad range of costs and effects of the interventions being compared and reports them separately. This encourages the analyst to present disaggregated measures of all types of health and non-health impacts across the different care sectors in line with the impact inventory [33]. The decision maker can then choose which costs and consequences are most relevant to their local context and viewpoint. In the absence of the usual cost-per-QALY assessment compared with NICE’s willingness-to-pay thresholds, decision makers will need to choose their own weighting system (value judgements) to decide whether the benefits of the DHI are worth the additional costs [28]. For example, some decision makers may place higher value on the effects of the DHI in improving patients’ wellbeing, whereas others may be more interested in how the DHI improves efficiency in health care delivery.

There may be settings where other forms of economic analysis may also be appropriate. For example, it may be more straightforward to measure both costs and effects in monetary terms (cost-benefit analysis) of DHIs targeting population-level prevention. In this case, health and non-health benefits can be measured using standard economic tools such as willingness to pay/accept methods. This would involve asking patients about the perceived benefit of the DHI and their willingness to pay for the DHI. For instance, previous studies used online choice experiments to estimate the value users attached to digital tools, and found that they were willing to pay around US$17,000 for search engines, US$3500 for digital maps and US$1000 for video streaming services [36]. Furthermore, when the DHI is in its pilot stage, a budget impact analysis may be sufficient [35, 37]. Budget impact assessments can help determine the likely costs, both incurred and saved (benefits), as a result of implementing the DHI.

An Application to the Living with Covid Recovery Programme

This section illustrates a step-by-step plan for incorporating the methodological aspects discussed in the previous section into an economic evaluation of the Living With Covid Recovery programme. Full details of the programme are reported elsewhere [38]. The main study is in its initial stages and a full health economic analysis plan has not been developed yet; therefore, this provides a timely case study to illustrate the distinct features of DHIs and consider afresh how to appropriately accommodate these in the design and analysis of the economic evaluation.

Many patients affected by Covid-19 experience prolonged moderate-to-severe symptoms follow infection, particularly fatigue, breathlessness and anxiety [39]. These persistent symptoms, also known as ‘Long Covid’, may not be related to the initial severity of the disease, and hence are not limited to those who have been admitted to hospital [40]. While cognitive behavioural therapy and physiotherapy may be effective in treating some of these symptoms, implementing them on a large scale using existing health system resources is very challenging, if not impossible. The need for facilitating access to Covid-19 rehabilitation treatments by harnessing digital technologies has been recognised by the English NHS to enhance face-to-face health care [41].

The Living With Covid Recovery programme has been recently proposed to provide a digitally supported, remote rehabilitation programme for patients with Long Covid in England. This programme is currently being piloted and includes several components: (i) development of clinical pathways aligned with NICE guidelines, (ii) a clinician-faced digital dashboard that summarises information about Long Covid patients and (iii) a patient-facing mobile app (Living With Covid app) to provide tailored rehabilitation according to patient symptoms.

Step 1—Choice of Comparator

The proposed rehabilitation programme has been designed as a whole new service in the context of the Covid-19 pandemic, where traditional face-to-face services struggled to accommodate the large number of patients affected with Long Covid. In addition, due to the nature of the pandemic, there were real difficulties in delivering face-to-face rehabilitation services. This makes the choice of comparator challenging because it is unclear what the current rehabilitation pathway and its associated health service use consists of. In addition, the combination of face-to-face and digital rehabilitation services differs widely across NHS providers.

On the one hand, comparing the digital intervention with face-to-face rehabilitation is not very relevant because we anticipate ‘usual care’ to include some digital component, for example text messages, given the ongoing social distancing rules. On the other hand, a head-to-head comparison between the proposed intervention and another DHI is not feasible as currently there is no well-defined digital platform to enable this comparison. Given the difficulties in characterising current practice, a relevant comparator might be defined broadly as existing rehabilitation services, whatever that entails within each NHS provider, not supported through the novel component of the proposed programme—the Living With Covid app. This emphasises the importance of an impact inventory approach to help describe and measure the services that patients in the ‘usual care’ group actually get.

Step 2—Deciding the Study Perspective

Deciding the point of view that will be taken for the economic evaluation is important because it determines the breadth of costs and effects that need to be considered. Long Covid directly affects patients’ mental and physical health as well as their ability to return to work, maintain social relationships and general wellbeing. In this case, a broader perspective, such as the societal perspective, will be required to capture all the relevant costs and effects of the proposed intervention versus usual care. This aligns well with the impact inventory approach and enables the inclusion of all relevant costs incurred beyond the health care sector and non-health benefits. More specifically, a societal perspective would help detect costs shifting between the health and economy sectors (e.g. productivity losses/gains) and benefits related to improving patients’ empowerment and user experience. Given that this is a new condition, patients’ (and carers’) views on the health and non-health outcomes that matter most to them will be crucial for the evaluation.

Step 3—Identification and Measurement of Effects

Identifying the relevant effects to be included in the economic evaluations should relate to the purpose of the DHI. The central aim of the Living With Covid Recovery programme is to help individuals self-manage Long Covid symptoms in order to enable them to carry out usual day-to-day activities. The intervention is not concerned with ‘curing’ Long Covid but instead supporting individuals to deal with those symptoms, until they eventually fade away or patients adapt to them. As a result, important effects of the intervention are anticipated to go beyond those related to an individual’s health, such as the impact on the individual’s ability to resume pre-Covid daily activities.

In this case, standard patient-reported outcome measures (PROMs) alone, such fatigue, anxiety scores or generic health-related quality of life (e.g. EQ-5D) are unlikely to capture important benefits of the intervention. Outcome measures that more accurately capture the extent to which Long Covid limits individuals’ day-to-day activities are needed. One such measure is the Work and Social Adjustment Scale (WSAS) [42], which seeks to quantify the impact of the specific health condition (rather than general health status) on individuals’ ability to undertake different activities, including those related to work, leisure and relationships. Other relevant effects might include (i) productivity changes in health system, for example the app-supported programme may reduce time spent by staff assessing and reviewing patients; (ii) impact on patient empowerment and how capable they feel about managing Long Covid symptoms and (iii) overall patient satisfaction/experience with Long Covid rehabilitation care.

Step 4—Identification, Measurement and Valuation of Resource Use

The first step should be to determine whether or not the costs associated with the development of the DHI are ‘sunk costs’. The Living With Covid app is being built on an existing digital platform (‘Living With’) that supports patients with certain long-term conditions, including rheumatoid arthritis and cancer, in the English NHS. As such, any research and development costs incurred with the main Living With digital platform should be excluded from the economic evaluation of the Living With Covid Recovery programme. Any resources related to modifications required by the Living With Covid app should be considered, including new content and software features and further iterative development based on user feedback. These resources will need to be discriminated by the Living With digital company and unit costs taken from their services.

Maintenance costs are likely to be close to zero as any resources related to hosting, updating and security are likely to pertain to the general Living With platform. However, time spent engaging with the mobile app may be considerable, for example because PROMs are expected to be frequently collected (e.g. monthly). Engagement time is collected through the app and can be costed using minimal wage rates. We might also expect differences between the intervention and usual care in terms of staff time and training needed to assess and review patients (during face-to-face consultations). These can be captured via clinical records and costed using unit costs of Health and Social Care (HSC) [43].

Individuals receiving the app-supported rehabilitation programme may be associated with different patterns of service use, including both GP and specialist (e.g. physiotherapist) consultations and hospital outpatient visits. Information about service use can be collected through the Living With app relatively straightforwardly, avoiding the need for additional patient questionnaires. National tariffs of NHS services and HSC unit costs can be applied directly to this service use.

An additional relevant cost component of the proposed intervention concerns the potential productivity gains or losses related to either absenteeism (sick leave) or presenteeism (reduced productivity while at work). Absenteeism costs are often measured using the human capital approach, where the number of days off work are costed according to hourly rates (in this case, sick leave pay rate). This also applies to partial return to work (part-time) if individuals were working full-time before getting ill. Presenteeism can be captured through the WSAS questionnaire, which asks individuals the extent to which Long Covid has limited their usual work activities. The number of work days lost can be obtained by multiplying the level of presenteeism (between 0 and 1) by the number of working days during a certain period (e.g. monthly).

Step 5—Economic Analysis

Given the points discussed in steps 2–4, a CCA that comprehensively reports all the costs and effects separately and in a disaggregated way is likely to be appropriate. This would facilitate (i) the economic evaluation to report on a wide range of non-health benefits of the programme, for example captured by the wide range of PROMs, which cannot be combined into a common unit of effect, (ii) the health commissioner/payer to choose the costs and effects that are most relevant to their decision context; for example, local commissioners may take a narrower perspective and exclude productivity costs; (iii) the decision makers to apply their own value judgements to decide whether the benefits of the intervention are worth the additional costs. For example, allowing individuals to return to work faster may be valued higher than improvements in Covid symptoms per se.

Given that the study is expecting to collect data on more generic measures of health, such as the EQ-5D, the cost effectiveness of the Living With Covid Recovery programme can also be reported using the standard cost per QALY (cost-utility) analysis. This may be useful to understand the programme’s value for money should the Living With Covid app be funded by the NHS. However, the QALY measure may fail to capture important (health) effects of the intervention (as explained in Step 3), and hence may underestimate its value for money.

Methodological Recommendations and Areas for Further Research

This paper discusses distinct features of DHIs compared with other health technologies, and their potential implications for the methodological choices underpinning the economic evaluation. This is a rapidly emerging area in health economics and we draw considerably on our own experience with many economic evaluations of DHIs [22-27] and development of methods guidelines [9, 28]. We highlight four methodological considerations for future economic evaluation studies that may help accommodate the distinctive nature of DHIs.

The economic evaluation should account for the rapidly evolving nature of DHIs at the design stage. This will typically involve (i) identifying a priori the different development stages of the DHI, (ii) discussing potential implications for resource use of the different DHI components and (iii) developing flexible data collection tools that can accommodate sudden impacts resulting from product evolvement.

An impact inventory that considers a broad range of health and non-health impacts, particularly those outside the health care sector, should be routinely included. The key advantage of the impact inventory is that it forces the analyst to set out explicitly what impacts and sectors should be considered in the economic evaluation.

A wider perspective than that of the national health service (e.g. societal) should be considered as default. This is crucial to enable important non-health benefits and costs that fall outside the health care sector to be included in evaluation of DHIs. Narrower viewpoints, such as that of the health service, can be included as a subset of the wider perspective.

A cost-consequence analysis should be adopted in the reference case analysis. This encourages a comprehensive, transparent description and comparison of the costs and effects between the relevant interventions. It enables the decision makers to use their own value judgements to decide whether the benefits of the DHI are worth the additional costs. A cost-utility analysis, often taking a health service perspective, may be undertaken complementarily if cost-per-QALY assessments are required, for example if the DHI requires a major financial commitment by the health service.

We recognise that there is considerable scope for variation regarding the methodological choices underpinning economic evaluations of DHIs. However, we hope that these recommendations provide a sensible starting point for encouraging further discussion about appropriate methods in this area. With this in mind, we have identified a few areas for further research.

First, greater understanding is needed about how much data the DHI user is required or prepared to share to achieve a certain health or non-health effect. DHIs offer scope for collecting personal data in great detail, which may add value in terms of effectiveness of the DHI, but tools for measuring such impacts require further attention. For example, appropriate ways of collecting user time spent with DHIs and how best to value that time has received little attention [44]. A related point is the role of data linked to the interactivity of use of DHIs. Process measures can be incorporated into DHIs to indicate the intensity (quantity) of this interactivity. Further work to estimate the opportunity costs of different levels of interactivity linked to effectiveness is warranted.

Second, DHIs are at a crossroad in terms of being low-cost interventions provided by the health service, or becoming commercial products involving either high prices, or exchange of much deeper health-related data than has been the case. Concerns about confidentiality and new General Data Protection Regulation (GDPR) requirements, especially of more detailed health-related data, may pose limits on the ‘barter’ model (no user charges in exchange for data), but perhaps less so if provided by the health service. There is ample scope to further explore the extent to which users might be willing to pay or be paid to use DHIs at different levels of data sharing.

Third, many DHIs are likely to be compared with interventions involving face-to-face interactions between health professionals and patients. Further assessment of the role of health professionals in augmenting the effectiveness of DHIs is required. As highlighted in Sect. 2, human support/facilitation has direct implications for the cost effectiveness of DHIs because the associated cost increases with additional users.

Fourth, the effects of DHIs, as demonstrated in clinical trials, have generally been of comparable sizes to those of relevant pharmaceuticals [4, 5]. However, the level of uncertainty and heterogeneity associated with the impact of DHIs tends to be much wider, which requires further attention. Firstly, the causal pathway tends to be more complex because it is a function of how the user interacts with a DHI and the wider health system. Secondly, the role of heterogeneity in the evaluation of DHIs is not well understood. For example, DHIs often involve interacting components and hence there can be significant heterogeneity in the way they are delivered. In addition, the usability of DHIs may differ according to important socio-demographic factors, such as age or socioeconomic status. Moreover, the settings (e.g. care pathways) in which the DHI is incorporated can be heterogeneous and affect the efficiency of the DHI.

Conclusion

The field of DHIs remains at a relatively early stage of development, which raises significant challenges for their economic evaluation. This paper has outlined important methodological considerations in the economic evaluation of DHIs, while highlighting areas that require further attention.

The distinct challenges posed by digital health interventions (DHIs) for economic evaluation are not well understood.

Compared with standard technologies such as drugs and medical devices, digital technologies tend to evolve faster over time, require active user input, interact more dynamically with user and environment, have distinct pricing and lead to diffused non-health impacts.

This can have important implications for the economic evaluations of DHIs with respect to the choice of comparator, study perspective, measurement of costs and effects and type of economic analysis.