'What You See is All There is': The Importance of Heuristics in Cost-Benefit Analysis (CBA) and Social Return on Investment (SROI) in the Evaluation of Public Health Interventions.

Health economists are currently debating, with some suspicion, the relative merits of cost-benefit analysis (CBA), grounded in theoretical welfare economics, and the proliferation of social return on investment (SROI), a pragmatic approach of developing a triple-bottom line (social, environmental and financial), but not grounded in welfare theory. We argue, in rather existential terms, that there is a need to understand the role of heuristics, or prior beliefs, in current 'best practice' in CBA and SROI. A taxonomy of CBA and SROI is presented, which summarises the origins of the methods, reporting guidance, publication checklist of quality of reporting, who is wanting these analytical approaches, and policy decision rule present. We argue that a bottom-up SROI is best thought of as localised CBA, building stakeholder involvement right into the framing of SROI, perhaps addressing or mitigating the effects of prior heuristics in top-down CBA. Behavioural CBA and social CBA recognise that people are not rational and that sources of value other than willingness to pay may best reflect social values. Standardisation of SROI and comparison with CBA may illuminate the role of prior heuristics and seek to better reflect social value in weighing up the costs and benefits of public health interventions at both a local and societal level.

Key Points for Decision Makers

Introduction

Cost-benefit analysis (CBA) is an economic method of analysis used to assess the costs and benefits of an intervention where both are expressed in monetary units. CBA focuses on the overall efficiency or value-for-money of one intervention compared to another [1]. Social return on investment (SROI) developed from traditional CBA and social accounting. SROI is an outcomes-based measurement tool that helps to understand and quantify the social, environmental and economic value being created from an intervention, and gives a ratio that states how much social value (in GBP [£]) is created for every £1 of investment [2]. Growing interest in the use of techniques of economic evaluation, such as CBA and SROI, in the evaluation of public health interventions to improve population health and reduce risk of future ill-health and disability is underpinned by the paradigm shift from ‘health services economics’ to ‘health economic evaluation’ [3, 4]. Public health initiatives to prevent ill-health and reduce health inequalities across the population life course often take place outside of traditional health care settings, for example the National Health Service (NHS) in the United Kingdom (UK). Many public health interventions take place in homes, schools, workplaces, and the community, creating complexity in economic evaluation [5]. This interest in applying economic evaluation methods to public health interventions takes us into existentialist considerations, not usually the subject of health economics. Existentialism is a philosophy rather than a scientific theory. It is a philosophy concerned with human beings as being ‘in the making’ [6].

This paper explores the philosophical issue of ‘how we think about and see the world’, that is, perspective and scope of evaluation, how we gather information from relevant stakeholders as society as a whole or individual stakeholders at a local level, and how we measure costs and outcomes in a wide or local public health evaluative space using heuristics (which we can think about as ‘rules-of-thumb’). Heuristics can lead us to attainable answers to difficult questions, which may not be the ‘right’ or most socially desirable answers, but that use the information and evidence we have to hand within a manageable timescale. Growing interest in public health economics through a paradigm shift from ‘healthcare economics’ to ‘health economics,’ allowed for a broader perspective of health and causes of inequalities in health to be considered by health economists [4].

The National Institute for Health and Care Excellence (NICE) [7] recommends the use of CBA and cost consequence analysis (CCA) in the evaluation of public health interventions, the latter being a disaggregated method. Many public health interventions represent good value for money with respect to the NICE threshold of £20,000 to £30,000 [8]. However, many commentators feel that the quality-adjusted life year (QALY) as an outcome measure in cost utility analysis (CUA) is not capable of capturing the wide range of relevant outcomes (often across a number of sectors) with respect to public health interventions that aim to improve the health of a population [5].

It is worth pausing to think about the underpinning ideas relating to such methodological debate. Daniel Kahneman’s [9] book Thinking, Fast and Slow, which explores the presence of two systems, System 1 (fast, automatic, frequent, emotional, stereotypic, unconscious) and System 2 (slow, effortful, infrequent, logical, calculating, conscious), has implications for the current debate amongst health economists about the similarities and differences between ‘top-down’ CBA, underpinned by welfare economics, and ‘bottom-up’ SROI, which is emerging from the field of social value calculation [10, 11]. These two methods of analysis, along with variants such as social CBA and behavioural CBA, are gaining more attention with respect to the evaluative space in public health economics [5, 12, 13]. Kahneman described a cognitive bias, which he termed ‘what you see is all there is’ (WYSIATI), that we use when we try to automatically make sense of partial information in a complex world. Kahneman [9] describes how, “Much of the time, the coherent story we put together is close enough to reality to support reasonable action” (p. 87). Heuristics are a type of problem-solving method that allows good-enough solutions to be made when decisions need to be made quickly, particularly when there are complex data. Neth and Gigerenzer [14] argued that heuristics are the answer to Simon’s [15] question: “How do human beings reason when the conditions for rationality postulated by the model of neoclassical economics are not met?” (p. 377).

Health economists have looked on as interest in, and the practice of, SROI has increased over the last 10 years with some adopting the method [16, 17]. Hutchinson et al [11] explored the extent to which academics in the health and social care research field, including health economists, have adopted SROI methods and assessed the quality of the papers they found according to a purpose-made checklist. From 868 papers screened, Hutchinson and colleagues identified only 8 studies that met the criteria for inclusion in their systematic review. Of these eight studies, Hutchinson and colleagues found study quality to be highly variable. In general, relatively high consistency and clarity were observed in the reporting of the research question, reasons for using this SROI methodology and justifying the need for the study. The authors found weaknesses in justifying stakeholders, reporting sample sizes, undertaking sensitivity analysis, and reporting unexpected or negative outcomes. They argued that there was little evidence that academic researchers in the health and social care field had advanced SROI methods or compared them with CBA methods beyond the Nicholls et al. [18] paper. As well as peer-reviewed published papers reporting the findings of SROI studies, examples of grey literature reports of SROI of third sector (i.e., voluntary sector) reports can be found on the Social Value UK [19] website.

This paper is, to our knowledge, the first attempt in health economics to draw out the similarities and differences in CBA and SROI methods and offer a guide to where they might be best applied in the evaluation of public health interventions. The paper builds on a very useful paper drawing out the similarities and differences in a US non-profit social enterprise setting [20]. The remainder of this paper is structured as follows. In the following sections we revisit the QALY paradigm, extra welfarism and public health; review the principles of cost-effectiveness analysis (CEA) and associated decision rules; explore the similarities and differences between CBA and SROI; compare sources of monetary values used in CBA and SOI, and finally stop to think about the precautionary principle in public health.

Revisiting Familiar Territory: Cost-Effectiveness Analysis and Decision Rules

NICE [21] recommends the use of CUA in the evaluation of most new drugs and technologies. This is derived from the evidence-based medicine paradigm in which “well-conducted RCTs can virtually eliminate biases and establish the direction of causation of the effect of an intervention” (p. 2) [22]. Randomised controlled trials (RCTs) use objective probabilities akin to base probabilities in the work of Kahneman [9] in Thinking, Fast and Slow. These are distinct from subjective probabilities, which can lead to bias in forming conclusions, which are arguably present in the use of heuristics or rules of thumb. In its Guide to the methods of technology appraisal, NICE [23] recommends that an intervention’s effectiveness should be established from external evidence with sufficient certainty. In their 2018 manual for guidelines in public health, NICE [21] broadens the evidence-based medicine criteria to allow subjective measures of effect, following the Bradford Hill criteria for judging causality. In public health, this broadening is necessary where trials are often underpowered or necessary evidence does not exist, otherwise trials would have to be extremely large and prohibitively expensive, leading to the use of evidence from natural experiments, non-controlled studies, and expert opinion [21].

In the extra-welfarist QALY paradigm, the additional cost of an intervention compared with usual care is divided by the quality-adjusted survival effects of the intervention (over and above those of usual care) to produce an incremental cost per QALY (also known as an incremental cost-effectiveness ratio [ICER]) [22]. The underlying assumption is a societal goal of QALY maximisation. Fischer and Ghelardi [22] have argued that decision makers using CEA are risk neutral, rather like large insurers pooling risk. This means they use a decision rule of whether an ICER for a particular intervention falls above or below the NICE threshold regardless of sampling variability. Fischer and Ghelardi questioned whether national health systems such as the NHS in the UK are in fact risk neutral or could be risk averse and pay more in the case of potential public health crises. This has undoubtedly been borne out during the COVID-19 pandemic. For example, in response to the expected surge of patients diagnosed with COVID-19, the NHS undertook several measures to ensure sufficient bed capacity, including block-buying capacity in independent hospitals, and the Chancellor of the Exchequer stated in Parliament that, “Whatever extra resources our NHS needs to cope with coronavirus—it will get” [24]. The decision-theory rules of CEA break down in such circumstances, as they assume risk neutrality and the presence of a budget envelope [25].

Back in 1992, Birch and Gafni [25] explored the theoretical underpinnings of CEA and CUA, and whether they do indeed meet societal goals. Birch and Gafni [25] argued that CEA, operationalised through CUA, is inconsistent with welfare economics objectives. They put forward alternative strategies for achieving such welfare economics objectives. Using quotes from Alice in Wonderland by Lewis Carroll, their paper asked whether this information used in economic evaluation, in line with current guidance, leads us to our chosen objectives (or where we want to be). They argued the need to examine whether CEA and CUA operationalise meeting defined societal goals. They argued that if “the objective is to evaluate programmes from a welfare economics perspective, then the use of cost-effectiveness or cost-utility methods should be applied in the context of integer programming problems [a heuristic programming technique]. Alternatively, welfare economic goals could be pursued by adopting a higher level of economic evaluation, i.e., cost-benefit analysis” (p. 295) [25]. CBA, unlike CEA and CUA, has its foundations in welfare economics.

Birch and Gafni [25] essentially were exploring the fact that CEA does not take account of societal or organisational budget constraint in itself. They argued that simple application of this cost-effectiveness ratio to decide between interventions “need not lead to the maximization of benefits from a fixed resource pool, nor to minimizing the cost of achieving a given objective” (p. 284) [25]. Essentially, without a budget constraint we ignore the principle of opportunity cost. This is an idea that was championed by Karl Claxton prior to the COVID-19 pandemic. He argued that the NICE threshold range of £20,000 to £30,000 per QALY, which has been implemented since 2004, is not based on evidence, and instead, a threshold of £13,000 per QALY should be used as it better reflects the opportunity cost of producing an additional QALY in the NHS [26]. Today, we are familiar with the concept that extra-welfarist, rather than welfarist, principles underpin such decision rules by NICE, that is, that of QALY maximisation [27].

Birch and Gafni [25] argued that, in theory, CUA was consistent with welfare economics theory of production and product mix using a QALY production frontier model, but that the decision rules of CUA recommended in the methods literature did not address the issue of allocative efficiency (i.e., how we achieve specified societal goals). They argued that for all this to work out, we need to assume that 1) the objective is to maximise QALYs produced from a given resource pool, and 2) the value of a given change in health status is assumed to be of equal value to society, irrespective of who receives it [25]. This extra-welfarist paradigm is different from the welfarist paradigm which acknowledges heterogeneity in willingness to pay (WTP) (compensating variation for a unit of health gain). We return to this WTP issue later in the paper.

Exploring the Similarities and Differences Between Cost-Benefit Analysis (CBA) and Social Return on Investment (SROI)

Below are brief descriptions of a number of methods of economic evaluation which are increasingly being advocated and applied in the evaluation of public health interventions. As stated above, with respect to the evaluation of public health interventions, with potential costs and benefits spanning a range of sectors beyond the health sector, and with multiple policy goals, over the last decade NICE has recommended the use of CBA and disaggregated CCA, and developed a range of ROI tools and explored the use of SROI [7, 13]. The following subsections complement our taxonomy of CBA and SOI shown in Table 1, which summarises the origins of the methods, reporting guidance, publication checklists of quality reporting, who is wanting these analytical approaches, and policy decision rules present.

Table 1

The taxonomy of sameness and difference for ROI, social ROI, CBA, social CBA and behavioural CBA

Approach	Origins	Reporting guidance	Publication checklist of quality of reporting/critique	Who is wanting these analytical approaches	Policy decision rule present
ROI	F. Donaldson Brown developed the expanded ROI measure, or DuPont formula, in 1914	Public Health England [28]	NICE [13]	e.g., local authorities and their public health commissioners in England, Public Health Wales, and Public Protection Scotland	Positive ROI or highest ROI
SROI	Developed in 1996 by REDF in the USA Developed from traditional CBA and social accounting	Nicholls et al [18] NEF Consulting [2]	Fujiwara [29, 30] Hutchinson et al. [11] Banke-Thomas, Madaj, Charles, and van den Broek [31]	e.g., local authorities and their public health commissioners in England, Public Health Wales, and Public Protection Scotland; the third sector (i.e., voluntary sector)	Positive SROI ratio
CBA	French engineer and economist Jules Dupuit introduced the concept of CBA in 1848; later formalised by Alfred Marshall, and developed extensively by Mishan [1]	McIntosh, Clarke, Frew, and Louviere [31]	Sanghera, Frew, and Roberts [32]	e.g., the UK Treasury	Positive net benefit or highest net benefit
Social CBA	Little and Mirrlees [34] and UNIDO developed social CBA techniques in the late 1960s	HM Treasury’s [35] The Green Book	HM Treasury’s [35] The Green Book	e.g., the UK Treasury	Positive net benefit or highest net benefit
Behavioural CBA	Behavioural economics theory [36, 37]	Hölzinger and Grayson [38]	Not available	e.g., the UK Treasury and the Behavioural Insight Team	Positive benefit cost ratio or highest benefit cost ratio

CBA cost benefit analysis, HM Her Majesty’s, NEF New Economics Foundation, NICE The National Institute for Health and Care Excellence, REDF Roberts Enterprise Development Fund, ROI return on investment, SROI social return on investment, UNIDO United Nations Industrial Development Organisation, USA United States of America

Return on Investment (ROI)

ROI is a performance measure used to evaluate the efficiency of an investment or compare the efficiency of a number of different investments. It tries to directly measure the amount of financial return on a particular investment relative to the investment’s cost. To calculate ROI, the net benefit (or return) of an investment is divided by the cost of that investment. The result is expressed as a percentage or ratio. ROI analysis has been advocated and is being used in public health to quantify the impact of recent disinvestment in public health, during a decade of austerity and coinciding with a move out of the NHS and into local authority in England, UK [39]. A recent review with 52 included studies found a median ROI of 14.3–1, and a median cost-benefit ratio (CBR) of 8.3. The median ROI for all 29 local public health interventions was 4.1–1, and a median CBR of 10.3. Even larger benefits were reported in 28 studies analysing nationwide public health interventions; the median ROI was 27.2, and median CBR was 17.5 [39]. The authors interpreted these results as demonstrating public health interventions as highly cost saving and that such cuts in spending on public health services represented a false economy. There were a handful of studies with a negative ROI or CBR in this review, for example Nichol [40]. To address the limitation of defining what constitutes a ‘public health intervention’, Masters et al used Acheson’s [41] broad definition of public health when considering their search strategy. By including the various fields of public health, the authors establish that these 52 included studies are representative of public health interventions in general.

Public Health England [28] have published resources to help local commissioners achieve value for money by estimating the ROI and cost effectiveness (cost per QALY) of public health programmes, including how to:

Assess which interventions provide the best value for money, by calculating their costs, benefits and ROI

Make the most of a budget by deciding how to split resources across different public health programmes

Compare costs, savings and clinical outcomes

Public Health England [28] have produced interactive tools for the following health conditions and services: older adults; colorectal cancer; NHS Diabetes Prevention Programme; end-of-life care; weight management; oral health in pre-school children; mental health service; musculoskeletal conditions; movement into employment; falls prevention; Best Start in Life; air pollution; contraceptive services, and cardiovascular disease. For example, the development of the older adults’ NHS and social care ROI tool presents ROI for four different analytical perspectives: NHS financial ROI (where benefits are measured exclusively as gross NHS savings for every £1 spent by commissioners on the intervention); social care financial ROI (where benefits are measured exclusively as gross social care savings for every £1 spent by commissioners on the intervention); financial ROI (where benefits are measured as gross NHS and social care savings for every £1 spent by commissioners on the intervention), and SROI (where benefits include gross NHS and social care savings in addition to monetised QALYs for every £1 spent by commissioners on the intervention) [42].

Social Return on Investment (SROI)

SROI was originally developed by the Roberts Enterprise Fund (REDF) in 1996 [43]. Since then, there has been a gradual revision of the original methodology, which has led to the ‘triple bottom line’ approach, underpinned by blended value accounting theory [44-47]. SROI is an evaluative framework that helps organisations to quantify the social, environmental and economic value being created by producing a ratio that states how much social value (in monetary terms) is created for every £1 of investment [2]. In the UK, SROI has been championed by the New Economics Foundation (NEF). NEF offers consultancy and training on the methodology and practice needed to conduct an SROI analysis to clients such as local authorities, universities, public health services, charities, and community groups. A key principle of SROI is understanding how and what changes occur as a result of an intervention (e.g., social, environmental or economic). This guides the approach to selecting which outcomes to measure and value (in order to provide evidence that change has taken place). Social prescribing, also known as community referral, is a way for health professionals to refer people to local, non-clinical services (such as nature-based activities) to support health and well-being [48]. SROI is increasingly being used as a method for analysing social prescribing interventions as it takes into account ‘well-being’ impacts into the analysis [49]. Social prescribing is an example of how SROI can be applied to measure localised CBA.

Cost-Benefit Analysis (CBA)

CBA was formalised in the US Flood Control Act of 1936 [50], which asserted that the federal government had a responsibility to support flood control activities “if the benefits to whomsoever they may accrue are in excess of the estimated costs, and if the lives and social security of people are otherwise adversely affected” (p. 71). The Kaldor-Hicks criteria of hypothetical compensation argued that a public policy was justified if it produced social gains in excess of social losses so that it was possible for winners from the policy to compensate losers even if compensation did not actually take place—the foundation of new applied welfare economics from the 1950s onwards. Hicks [51] argued that the business of the economist is “to estimate as far as he [or she] can the gains and losses that are likely to accrue, to various classes, or sections of the population, from the proposed action” (p. 366) with a view to hypothetical compensation being possible. Adler and Posner [52] rejected the Kaldor-Hicks approach, emphasised the social objective of improving overall well-being with considerable emphasis on distributional issues, for example, by statistical averaging of group gains and losses over time and space. There is growing interest particularly in environmental economics in the inclusion of equity weights in CBA and that it may make a considerable difference to policy recommendations. However, economists applying mainstream CBA have been reluctant to use such weights because of the uncertainty about their values and the wide range they could take [53].

Social Cost-Benefit Analysis (CBA)

Little and Mirrlees [34] and the United Nations Industrial Development Organisation (UNIDO) developed social CBA techniques in the late 1960s. Social CBA is a marginal analysis technique that assesses the impact of different options on social welfare [35], taking into account the social, economic, environmental and financial impacts. This technique can be used to express the social costs and social benefits of government policies to society in monetary terms so that the consequences of a range of policies can be compared using a common metric. Change in welfare is measured as the aggregate WTP for the effects of a certain policy intervention. Social CBA has been championed in the Netherlands with respect to the evaluation of transport and road system planning including congestion charges, and tobacco and alcohol pricing and policy [54-56].

Behavioural Cost-Benefit Analysis (CBA)

CBA is based on the neoclassical welfare economics paradigm, which assumes that people maximise their long-term best interest, have stable preferences, and are consistent rational actors. Behavioural CBA involves bringing insights from behavioural economics into CBA to reflect the fact that people actually do not always behave rationally [57]. Behavioural economics can provide an empirically informed perspective on CBA, including the important realisation that even subtle features of the environment or the design of public health interventions can have meaningful impacts on behaviour [58]. As such, behavioural economics underpins the ‘nudge theory’, which argues the case for positive reinforcement and indirect suggestions as ways to influence the behaviour and decision making of groups or individuals [59].

With respect to the economic evaluation of public health interventions, it has proved challenging sometimes to capture economic benefits, for example, CBA of the Sure Start Programme in the UK for young families [60]. Public health interventions can fail to be effective due to assumptions about rationality, which may be limited or bounded as a result of limited willpower, addiction, social norms, and the context in which choices are made [58]. The behavioural economics approach, applied to CBA, draws on related fields of psychology and neuroscience.

Behavioural economics has implications for the way we conduct CBA as a result of this, particularly whether we should actually use WTP estimates as a basis for shadow prices where no market values exist, for example, in the case of a public good such as prevention. Asking people to state their preference for what they would pay for a hypothetical public health intervention to which they may not respond rationally may lead to misleading non-market values which could influence CBA calculations. This applies equally to the argument that a distinction should be made between decision-utility and experienced-utility, making the normative assumption that the allocation of resources should be based on experienced-utility rather than decision-utility. This approach favours prospect theory as a basis for CBA because decision making under uncertainty departs from expected utility theory [61, 62]. Prospect theory, developed by Daniel Kahneman and Amos Tversky in 1979, underpins behavioural economics and aims to describe the actual behaviour of people and challenges the expected utility theory developed by John von Neumann and Oskar Morgenstern. Prospect theory involves individuals weighing up their potential losses and gains of an action and does not require the assumptions of expected utility theory, that is, rationality [61, 62]. This has implications for what discount rates are chosen in CBA because of present-bias, due to irrational impatience. Overall, behavioural economics cautions health economists to the framing of WTP questions and how these might influence the outcome of applying CBA in the evaluation of public health interventions where health and other benefits are often accrued far into the future. Field experiments are becoming more common beyond laboratory experiments to test out whether irrational behaviour as found in the laboratory would influence the calculation of shadow prices representing people’s behaviour in real life. This is an effort to use evidence to improve the conduct of CBA [57]. The UK government have recognised the importance of behavioural economics in establishing the Behavioural Insights Team (https://www.bi.team/).

Comparison of Calculation Methods

In Box 1 below we set out the formulae for the calculation of ROI, SROI and benefit cost ratios.

Box 1 Calculating return on investment (ROI), social ROI (SROI) and cost-benefit analysis (CBA)

In theory, the methods discussed above in Box 1 all share the stages of identifying, measuring and valuing the costs and benefits, but they may vary in perspective, range and the extent to which they require all items to be monetised.

Sources of Value for Cost-Benefit Analysis (CBA) and Social Return on Investment (SROI)

In CBA, where there is no market price, analysts have used stated preference WTP, also known as contingent valuation (CV), as a means of gaining a shadow price [32]. Compared with the number of cost-effectiveness and cost-utility studies undertaken as health technology assessment (HTA), there have been relatively few cost-benefit studies in health and social care and we find ourselves back in the 1990s when the idea of the use of CBA in health economics was first explored [63]. CBA has been used more widely in environmental economics and more recently SROI has been adopted by the third sector across the economy in the evaluation of social programmes [64-66]. In 1991, Gafni argued that WTP could be used both in a private individual and public policy setting to establish non-market values. His starting point was that the general question that CBA sets out to answer is whether one or a number of programmes should be undertaken at all and, if investable funds are limited, which programmes among those predicted to generate a surplus of benefit over costs should be selected [67]. This top-down CBA method applies to the welfare of the economy or society as a whole rather than with any smaller group within that economy or society (in this way, it is different from the bottom-up SROI method, which often has a more local commissioning focus). In line with the Potential Pareto Improvement Criterion, welfare is measured by an individual’s maximum WTP for the good or service gained as the measure of benefits or the minimum level of compensation for them to give up that good or service. This is based on the assumption of ‘rationality’, which links back to the more recent enquiry of behavioural economists as to whether such rationality exists. The WTP and contingent valuation literature has developed into the discrete choice experiment methodology over the last two decades, involving assumptions of rationality [68-70].

SROI is described as a form of CBA and can be thought of as a localised CBA, which aims to calculate the triple bottom line for a given geographical area population or defined group of stakeholders rather than the whole of the economy or society as in CBA. SROI often uses values from the growing source of social value estimates used in social well-being valuation, such as the Social Value Bank [10, 71]. These could arguably be used more widely for CBA but have not been to date perhaps on theoretical grounds.

The Precautionary Principle in Public Health

The precautionary principle emerged during the 1970s in German environmental law, where it was referred to as Vorsorgeprinzip. The precautionary principle enables decision makers to adopt precautionary measures when scientific information is uncertain and there is potential for causing harm. While science can improve our knowledge base, it can rarely provide certainty, and so decisions must be made on the basis of the best available information, while erring on the side of caution. We have seen the use of the precautionary principle internationally in the handling of the COVID-19 pandemic [72, 73]. The precautionary principle in relation to the evaluation of public health interventions, as put forward by Fischer and Ghelardi [22], reverses the onus of proof of effectiveness and cost effectiveness of interventions designed to reduce harm and it seems particularly relevant at a time of the COVID-19 pandemic. The UK government acted without full evidence but aware that the direct health effects of COVID-19 on society would be amplified by the wider economic impacts on health resulting from economic hardship through (1) public and private indebtedness; (2) an uneven impact of lockdown across regions and countries, and (3) poverty and subsequent effects on health and access to health and social care [74]. Fisher and Ghelardi [22] state that, “The expectation in Public Health is that interventions employed to reduce harm will not actually increase harm, where ‘harm’ in this context does not include opportunity cost” (p. 1). This is contrary to the evidence-based medicine paradigm where the onus is to demonstrate clinical effectiveness and cost effectiveness with sufficient certainty before the introduction of a new drug or device. These authors consider the cost of society taking precautions to protect public health or mitigate the effects of poverty, for example, and that these precautions are not costless. The Rio Declaration on Environment and Development [75] states that, “In order to protect the environment, the precautionary approach shall be widely applied by States according to their capabilities. Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation” (principle 15).

The problem in applying the precautionary principle across different sectors is that there is no agreed definition about the degree of precaution and appropriate costs, who bears those costs and who may reap the subsequent benefits. It can be argued, as we attempt here, that CBA and SROI are actually more closely aligned to the decision theory Bayesian approach to the evaluation of public health interventions put forward by Fischer et al [12] than CEA based on RCT evidence. We argue that the use of a posterior combination of existing and new updated evidence from a range of sources as well as, or in the absence of, RCT evidence, and awareness of potential ‘bias’, along the lines of Kahneman’s [9] ‘what you see is all there is’ (WYSIATI) principle describes how analysts might take a viewpoint of society or a local community in the case of CBA and SROI. To this end, new information has value in that it can alter or improve decision making. Fischer et al argue that decision theory aims to forecast the likely impact of a public health intervention and that then it is up to policymakers to weigh up wider costs and outcomes beyond the direct health impacts desired. The processes of CBA and SROI really aim to capture these and place monetary values on them where possible.

Aware of the danger of bias, in a US non-profit setting of the evaluation of social enterprises, Cordes [20] alludes to the danger of thinking ‘fast’ using readily available values in a heuristic sense “because of difficulties in monetization, attempting to quantify non-profit performance with either SROI analysis or CBA will tend to favor activities with outcomes that are readily translated into dollars, and/or create incentives for non-profits to focus more on those activities that can be monetized” (p. 104).

Choice of Cost-Benefit Analysis (CBA) or Social Return on Investment (SROI) Methods

In more stable times, budget constraints require weighing up the costs and benefits of using scarce resources in different ways at both the societal and local level. Below is a guide for health economists to think about, which methods might be most appropriate to provide evidence for policy makers at a societal or local level (Fig. 1).

Fig. 1

The importance of heuristics in cost-benefit analysis (CBA) and social return on investment (SROI): a decision guide in public health

Recent research in the field of public health economics has identified that many health and local authority commissioners and their public health colleagues want to know is really a version of localised CBA, across sectors, which is why they have been drawn to SROI [76]. Where there is an appetite for shared budgets and joined up working at a local level, SROI and ROI evidence can make a meaningful contribution to the evidence base on which public health service commissioning decisions are made. However, it would be naïve not to acknowledge the influence of local politics and political horizons on decision making. Local stakeholders may still feel very restricted by the structure of individual departmental budgets and the political consequences of not prioritising the goals set within those individual silos.

Should We Use Multiple Methods of Analysis?

Ultimately, health economists are, or should be, concerned with the measurement of opportunity cost. This sometimes gets lost. Methods such as ROI and SROI are criticised because they do not measure or account for opportunity cost. CEA and CUA lend themselves to decision rules such as reference to payer thresholds. CBA, SROI and ROI facilitate rankings of benefit cost ratios. CCA is disaggregated leaving the decision maker to weigh up the relative merits of alternatives without a common denominator. It has been suggested that we should apply multiple methods of analysis, which at one level enables the capture of a wide range of costs and benefits, but we are left with the decision rule challenge. If one method of analysis provides evidence that leads to a conclusion that an intervention should be funded and another method contradicts that conclusion, relative to some decision rule, the decision maker or commissioner of services is none the wiser. If we choose to only use one method of analysis, then we need to be very clear and open about the heuristics or prior beliefs that underpin and bound the perspective and range of costs and benefits captured by that method.

Discussion

When addressing the question of whether a monetary value could be put on a human life in the early days of the development of CBA or CUA, Mishan described Broome’s scepticism as “faint-hearted intellectual doodling” [77-79]. We hope we have done a little better than that in this paper in exploring the similarities and differences between CBA and SROI methodologies and their use in public health economics today. CBA and SROI are heavily dependent on the heuristics and value judgements that underpin their perspective, range of costs and outcomes captured and valued, and time horizon chosen. As analysts, our choice of perspective, and hence the range of evidence included in our calculations, and indeed the sources of that evidence, determine the bias that shape our reality. We started with the quote from Daniel Kahneman [9], ‘what you see is all there is’ (WYSIATI), because it sums up the fact that we each, as researchers, define and interpret our own reality, and that this can greatly influence the answers we come to in our health economics analysis, and the subsequent use of such evidence by, for example, NICE or local commissioners. This subsequently shapes the services that end up being commissioned and thereby influences the health of the population at a local or national level.

Behavioural economists are showing us now that people are far from rational. This is important in the design and implementation of public health interventions to protect and promote population health. As health economists, we do need to revisit the assumptions of welfare economics before just accepting the results of CBA and SROI analyses. The SROI methodology has chosen to routinely source shadow prices and social values from HACT and the Social Value Bank, for example, in place of WTP used in CBA. These values are drawn from four large national UK datasets that include data on well-being and life circumstances: British Household Panel Survey [80]; Understanding Society [81]; the Crime Survey for England and Wales [82], and the Taking Part Survey [83]. These sources are aimed to be representative of the population as a whole; however, we have to acknowledge that there will be groups within the population with very different views. Social preferences are multimodal and there may well be distinct groups. An example of this would be current attitudes to receiving the vaccination against COVID-19.

Conclusion

The news is good, aligned with decision theory and the precautionary principle, the current debate over scrutinising and standardising SROI and comparing it to CBA may end up improving the usefulness of CBA and SROI as evaluative methods that recognise the role of prior heuristics. In the UK, measuring social value has become embedded in UK legislation, for example, the Public Services (Social Value) Act 2012 [84] requires public sector commissioners to factor in economic, social and environmental well-being in connection with public services contracts. Health economists using CBA and SROI methods are seeking to better reflect social value in weighing up the costs and benefits of public health interventions at both a local and societal level.

SROI is best thought of as localised CBA. A guide is proposed for deciding whether SROI or CBA is most appropriate in any evaluative situation.

The top-down CBA method applies to the welfare of the economy or society as a whole rather than with any smaller group within that economy or society (in this way, it is different from the bottom-up SROI method, which often has a more local commissioning focus).

When designing and implementing public health interventions to protect and promote population health, we need to remember that people are not rational, and instead be aware of heuristics (prior beliefs).

We need to use evaluative methods that recognise the role of prior heuristics that better reflect social value in weighing up the costs and benefits of public health interventions at both a local and societal level.