The Sandbox Approach and its Potential for Use in Health Technology Assessment: A Literature Review.

BACKGROUND: The concept of the regulatory sandbox-a safe space for testing new regulatory processes-was first used within the financial technologies (FinTech) sector, but has since expanded into other sectors, including healthcare.
OBJECTIVES: This review aims to describe the extent of use of sandboxes in healthcare and assess the potential for the sandbox approach to be used to test and develop emerging health technology assessment (HTA) methods, policies and processes for innovative technologies.
METHODS: A systematic literature review was undertaken to identify published papers and reports that described and/or assessed the use of sandboxes in the healthcare sector. Searches were conducted in Medline, Embase, Econlit, Social Policy and Practice, and Health Management Information Consortium databases from inception to March 2020. Free-text Google search was also conducted to identify relevant grey literature. Only papers and reports discussing or evaluating the use of sandboxes in healthcare settings and published in English were included. Included studies were qualitatively summarised using a thematic analysis approach.
RESULTS: Overall, 46 papers and reports were included. The topics covered were classified into 4 major themes: history of the regulatory sandbox, the sandbox as a testing environment, the sandbox as a regulatory approach, examples of using sandboxes in healthcare. Findings show that the use of regulatory sandboxes in healthcare is relatively new and primarily used in high-income countries to support the adoption of new technologies, particularly those related to digital health. Recommendations are made based on these findings to guide its use in HTA policy and methods development.
CONCLUSIONS: Sandboxes are increasingly used within healthcare regulation. Despite its potential, this approach has not been used in HTA policy and methodological developments to date. HTA agencies should consider this approach to facilitate developing policies, methods and processes for innovative and disruptive health technologies. Transferability to low- and middle-income countries' settings, however, should be assessed.

Key Points for Decision Makers

Introduction

The development of innovative products, services and business models is often delayed or prevented due to regulatory and governance restrictions; however, innovation is crucial in ensuring economic growth and societal benefit [1]. As technological advancements, such as the use of big data and artificial intelligence (AI), start to emerge, regulatory and health technology assessment (HTA) agencies are required to evolve to allow these potentially disruptive innovations to be implemented for the benefit of society [2-4].

A range of mechanisms are being adopted by regulators to support innovation while also delivering appropriate regulatory oversight, through what is referred to as “anticipatory regulation”, where regulation is seen as a support tool for safeguarding responsible innovation, rather than a barrier [5]. One such mechanism is live-testing environments such as ‘sandboxes’ [1]. These allow innovators to trial products, services, and business models in a safe space, to confirm their compliance with existing regulation before implementing them within the wider sector. The UK’s Financial Conduct Authority (FCA) first used the sandbox environment for exploring regulatory issues in 2016 [6]. This regulatory sandbox was described ‘a safe space in which businesses can test innovative products, services, business models and delivery mechanisms without immediately incurring all the normal regulatory consequences.

The sandbox environment also allows exploration of processes that may violate current rules and regulations but have the potential to reap a large benefit if introduced into standard practice [7]. These testing environments have also been referred to as ‘testbeds’ or ‘living labs’, which can lead to confusion around their definition within the literature [8].

This review aims to provide an overview of sandboxes and provide examples of their current use and applications within the healthcare sector. It also aims to discuss the potential of using these environments by HTA agencies for trialling innovative ways of assessment and funding new technologies, including AI-based digital technologies and advanced therapy medicinal products, before their widespread adoption and implementation [9].

Methods

A scoping review was undertaken to identify published papers and reports that described or assessed the use of sandboxes in the healthcare sector. The search strategy is presented in the Supplementary Material. It was run across the following databases simultaneously via Ovid Interface: Medline (1946–March 2020), Embase (1974–March 2020), Econlit (1886–March 2020), Social Policy and Practice (1890s–March 2020) and Health Management Information Consortium (1979–March 2020).

A similar search strategy was employed for search engine (Google/Google Scholar) use, mainly focused around “policy sandbox”, “regulatory sandbox”, “sandbox” and the various healthcare search terms listed in the search strategy presented in the Supplementary Material. The search was limited to articles published in English. We did not apply any publication date limits.

The search was conducted on 11th March 2020. The records identified were exported into an Excel Database and checked for inclusion by two reviewers (EL and DD) independently. The main inclusion criterion was that the article should focus on the definition or application of policy or regulatory sandboxes in the healthcare sector. These articles could be scientifically published papers or grey literature (government reports, organisational reports). Articles were excluded if they were published in a language other than English, related to sectors than healthcare or used the term sandbox in an unrelated context (e.g., children’s playgrounds). There was no time cut-off in the exclusion criteria. Any disagreements between the two reviewers were resolved through discussion and consensus building. Extracted information included country, sector, main stakeholders involved, aim of the sandbox, technologies/interventions tested, the year it was reported and its active status. The themes extracted from the included articles were mapped by a single reviewer (EL). These themes were then discussed with the second reviewer (DD) and a final list was agreed upon between the two. The extracted themes were then described and discussed narratively. The reviewers followed PRISMA guidelines for conducting and reporting of reviews.

Results

The database search identified 141 records. After deduplication, including articles identified from additional sources and the grey literature and applying the inclusion/exclusion criteria, a total of 46 articles and reports were finally included [1, 5–8, 10–33, 36–39, 44–52, 55–58]. The included articles and reports are described in more detail in the Supplementary Material. Figure 1 presents a flow chart of the search process. A diagram presenting the main themes identified in the included articles and reports is also presented (see Fig. 2). These themes are discussed in turn below.

Fig. 1

PRISMA flow chart of the literature review process

Fig. 2

Themes identified from the included reports and studies

History of the Regulatory Sandbox

The concept of “the regulatory sandbox”—was first considered in 2015 by Mark Walport, former chief scientific advisor of the UK Government, in the review ‘FinTech Futures’ [10]. The financial technologies (FinTech) sector is concerned with the ‘development and commercialisation of new financial business models and disruptive innovation’. However, as the sector also exposes new methods for business and consumer fraud and exploitation, it is key that both the government and regulators maintain an oversight of activity within the sector to mitigate any potential risk.

Walport recommended that the financial sector develop a similar system to that used in the pharmaceutical sector, where new biomedical approaches are assessed using clinical trials, allowing testing, and monitoring to be done in a safe, controlled environment. The review recommended that the financial sector develop a similar system, allowing new technologies and consumer propositions to be tested in a safe environment without the threat of destabilising current financial systems [10]. This system was referred to as a ‘sandbox’, where regulators, institutions and FinTech companies could collaborate within virtual environments, and even with real data.

Following these recommendations, the UK’s Financial Conduct Authority (FCA) published a report referencing the feasibility of introducing a ‘regulatory sandbox’ as part of its Project Innovate programme, which aims to challenge regulatory barriers to allow innovation within the financial sector [6]. The regulatory sandbox was described as ‘a safe space in which businesses can test innovative products, services, business models and delivery mechanisms without immediately incurring all the normal regulatory consequences’.

The FCA regulatory sandbox was launched in June 2016, with the framework piloted using a reduced number of businesses in the first year. Since then, the regulatory sandbox concept has moved beyond the financial sector and is now being utilised across multiple sectors across the globe. One area of interest that has emerged is how these live-testing environments can be used within health and social care settings. This potential use was first realised through the set-up of the Care Quality Commission’s (CQC) regulatory sandbox—the first regulatory sandbox for healthcare innovation in the UK [11].

The Sandbox as a Testing Environment

Arntzen et al [8] categorised testing environments into different groups depending on two factors: how controlled the environment is, and at which point the approach is used in the product development process. The diagram in Fig. 3, adapted from Arntzen et al [8], explains the key differences between common testing environments used in product innovation in terms of the level of control in the environment and the stage in development.

Fig. 3

The main testing environments used in product innovation (adapted from Arntzen et al. [8])

“Living labs” can be defined as “user-centred open innovation ecosystems based on a systematic co-creation approach” [8]. They operate in simulated or real-world environments during the earliest stages of product development, encouraging all stakeholders to collaborate early in the innovation process.

“Sandboxes” allow innovators to trial these products, services and business models in a safe space, to confirm their compliance with existing regulation before implementing them within the wider sector. These environments also allow exploration of processes that may violate current rules and regulations but have the potential to reap a large benefit if introduced into standard practice [7]. They operate further along in the product development process.

Finally, “real-world test beds” are controlled environments for testing innovative products in real-world conditions in the environments in which they will be used or operated. These are useful in the latest stages of product development when the product is close to implementation. Laboratories and simulated/constructed environments can also be classified as test beds but have varying degrees of real-world components compared to real-world test beds.

The Sandbox as a Regulatory Approach

According to Armstrong and Rae, Regulation can be split into three categories of approaches: advisory, adaptive and anticipatory [12]. A sandbox’s position within these categories is determined by several factors; its goal, its desired outcome and the participants involved in it (Fig. 4). Armstrong and Rae [12] described the sandbox approach as an example of “anticipatory regulation”, where regulation is seen as a support tool rather than a barrier to innovation. Anticipatory regulation is designed to anticipate future challenges, and support innovators when faced with them.

Fig. 4

Levels of regulation used during product innovation (adapted from Armstrong and Rae [12])

Advisory approaches aim to aid new products and services in meeting existing regulations. The outcome of these approaches is a change to the new product or service, and usually only involves the regulators and innovators.

Adaptive approaches are more flexible and aim to support innovation through the adaptation of current regulations to suit the new product or service. These approaches result in not only product changes, but also changes to policy and regulation.

Finally, anticipatory approaches are the most flexible, as these aim to develop regulation in an iterative process alongside the advancement of new products and services. The outcome of these approaches is often a more comprehensive understanding of regulatory requirements, and therefore requires input of a much more varied selection of stakeholders. The risks and uncertainty increase with each approach; however, anticipatory approaches are the most proactive and flexible, so may lead to increased improvement in the long term.

Anticipatory regulatory approaches can also be described according to six key principles: inclusive and collaborative, future-facing, proactive, iterative, outcomes-based and experimental [5]. For a sandbox to truly be anticipatory in its approach, it must consider all these principles within its intended activity and outcomes.

Examples of Using the Sandbox Approach in Healthcare

Sandboxes have the potential to be beneficial in driving forward improvement of health and social care services. In this context, the regulatory sandbox aims to not only improve healthcare experience and outcomes, but also to improve the experiences of healthcare providers, commissioners, and regulators. The sandbox participants collaborate to determine ‘what good looks like’, and to design and drive innovation without the associated risks [13]. The examples identified from the published papers and reports have been categorised as “outcomes-focused” or “data-focused”, depending on their aims and outputs. These sandboxes are summarised below and in Table 1 and described in detail in the Supplementary Material. These detailed discussions include examples of how the sandboxes have led to changes in related policy or processes.

Table 1

Examples of sandboxes identified through the review

	Location	Name	Organisation(s)	Aim	Year Reported	Still Active? (Y/N)	References
Outcomes focused	UK	HealthTech Regulatory Sandbox	ICO, Data Guardian, NICE	Cross sector collaboration for ensuring successful implementation into NHS	2018	Y	[14]
	UK	CQC Regulatory Sandbox	Care Quality Commission	Defining ‘good care’, and how to facilitate and deploy new innovations	2019	Y	[11, 15–17]
	UK (Scotland)	Care Inspectorate Sandbox	Care Inspectorate	Focus on restructuring social care services to suit evolving care system	2018	Y	[13]
	UK	NHS Test Bed programme	NHS	Removing barriers to uptake of innovative technologies within the NHS; test and evaluate the impact of digital innovations in real-world settings	2015	Y	[18–22]
	Singapore	Licensing Experimentation and Adaptation Programme (LEAP)	Ministry of Health	Promote an effective way of supporting innovation while maintaining patient safety and welfare, through improved interaction between industry and regulators	2018	Y	[23]
	Japan	Regulatory Sandbox Framework	Government of Japan	Facilitate development of innovative technologies and business models in Japan	2018	Y	[24]
Data focused	UK	ICO Sandbox Programme	ICO	Delivering real benefit for the UK public through innovations in technology and use of data	2019	Y	[25]
	UK	Health Data Research UK Sandbox	HDR UK	Virtual environment that offers access to large scale health data for the development of products, services or innovations that benefit the population	2019	Y	[26]
	UK	Kernow Health CIC Sandbox	Kernow Health CIC	Opportunity to sample and improve products in real-life NHS clinical environments	2019	Y	[27]
	UK	Digital Health Jersey Sandbox	Digital Jersey	Providing testbed environments to companies that are developing digital health solutions	2018	Y	[28]
	Multiple (Europe)	eIT Health Digital Sandbox	European Institute of Innovation and Technology	Improve access to registries, biobanks, and other digital health sources for SMEs	2019	Y	[29]
	USA	Digital Health Sandbox Program	Massachusetts eHealth Institute	Support product development within digital health companies and promote the use of sandbox environments to a varied user base	2019	Y	[30]
	USA	ROMOP and PatientExplore	University of California, San Francisco	Provision of sandbox servers consisting of synthetic patient data for users to test the tools and access synthetic data sources.	2018	Y	[31, 32]

CIC Community Interest Company, CQC Care Quality Commission, eIT European Institute of Innovation and Technology, HDR Health Data Research, ICO Information Commissioner’s Office, NHS National Health Service, NICE National Institute for Health and Care Excellence, ROMOP R-Observational Medical Outcomes Partnership

Outcomes-Focused

Sandboxes included under this category focus on optimising patient outcomes achieved from the adoption or roll-out of interventions or service models. The majority of the interventions tested were digital health-related such as digital triage tools, the use of machine learning (ML) applications within diagnostics, interventions related to tele-medicine and mobile health, and the use of blockchain technology. Additionally, sandboxes have been set up to test service models such as those focused on the restructure of social care services and determining the appropriate staff skill-mix to support care home services.

Data-Focused

While electronic health records (EHRs) are becoming more common within healthcare as healthcare systems adopt new technology, issues persist relating to access to EHRs for biomedical and clinical research. However, the real-world data within these records are invaluable in progressing research and development, and therefore solutions to these issues are paramount in ensuring these sources are utilised effectively [33]. As a result, a related sandbox solution has become popular with organisations using electronic health data to facilitate testing of new technologies. These organisations either offer a service whereby they facilitate access to health data or are involved in the design of software tools to facilitate access and processing of health data for product testing.

Discussion

Summary of Findings

This review identified 46 articles and reports that covered the principles, development and use of the sandbox approach in healthcare. A thematic analysis of these articles and reports identified four key themes covering the history and development of this approach, examples of its use in healthcare and applying it to facilitate regulating and access to innovations. The examples identified from the literature highlight the potential usefulness of this approach for developing and implementing innovations in HTA methods, processes and policies in response to new and disruptive technology development.

Benefits of the Sandbox Approach

The main benefits of the sandboxes identified in this review include reduction in time and cost of getting products to market as delays due to regulatory uncertainty have a larger impact on first-time innovators [6]. Increased access to finances for innovators is also reported, as innovation is reliant on sufficient investment, and companies can struggle to raise funds due to regulatory uncertainty. Increased throughput of tested and introduced products to market is another important benefit, as regulatory uncertainty prevents the most innovative products from reaching the market, as they are often abandoned at early stages due to associated risks.

Improved collaboration between the regulator and innovators to ensure protection of consumers is also an important reported benefit. The sandbox approach has also been applied in numerous other sectors aside from FinTech and health, including energy and transport [34, 35]. The aim of these sandboxes differs between countries; for example the Singapore Energy Market Authority’s sandbox seeks innovations with the gas and electricity space, whereas the equivalent energy sandbox in Germany is focusing on increasing use of renewable energy sources [1]. These sectors similarly commented that the sandbox approach helped innovators and businesses to trial novel products, services and business models, to ensure viability before marketing. These benefits are equally applicable to HTA, given its pivotal role as a pre-requisite for access to innovative technologies in many countries, where early engagement with innovators can facilitate faster adoption of these technologies [3].

Approaches and Requirements for Developing a Policy Sandbox

Previous and ongoing sandboxes have yielded useful information on what approaches work when facilitating a sandbox programme [36]. One of the key measures identified for the success of sandboxes in general is stakeholders’ ability to collaborate effectively. This collaboration can be assisted using directed consortia, a partnership framework that provides collaborative environments for multiple, and sometime competitive, organisations with differing priorities [37]. Providing a neutral consortium environment facilitates this collaborative working. The problem of stakeholder engagement can also be seen in healthcare, whereby innovative technologies or products are often met with scepticism. Clinicians and patients may favour what is familiar to them rather than choosing to test new technologies. Thus, stakeholder engagement, including public and patient involvement, will be key in developing these innovations to ensure its uptake and diffusion within the healthcare system.

Consortia are often used during biomedical product translation. Research by consortium is also fast becoming a permanent fixture within biomedical R&D, with potential to be adapted for use by other areas of the healthcare sector [38].

The use of clear mission statements, targeted deliverables and timeframes, workstreams defined by participants involved, and expected milestones are key success factors [37]. Governance structures are used to define strategic decision making, with transparency being key to ensuring good collaboration and dissemination of results. Use of these features would provide clarity to those involved regarding their responsibilities and expectations during the testing process, as well as facilitating effective communication internally and externally. The use of eligibility criteria for access to the sandbox is also very important [36].

Potential Application of Sandboxes in HTA

While there is a selection of sandboxes focused on regulatory and data access challenges, there is limited sandbox activity that relates directly to HTA. Some projects currently being explored within the identified regulatory sandboxes are directly relevant to the HTA perspective, as they are considering advancements in innovative health technologies that do not currently have an established appraisal pathway within HTA agencies. Many of these are related to the advancement of digital technologies for diagnostics, disease management and prognostics such as the UK CQC sandbox, the UK NHS Test Beds Programme and the Health Data Research sandbox [11, 18, 26].

As digital health technology advances, HTA agencies must adapt to successfully appraise such technologies. NICE has published evidence standards relating to digital health technologies as part of a project funded by NHS UK [39]. The aim of the standards is to improve innovator and commissioner understanding of the levels of evidence required for digital healthcare technologies. While a sandbox has been used to explore regulation of these products in the UK, sandboxes have not been used to determine the best methods for assessing these products from an HTA perspective or the policies and processes that underpin these assessments. Developing these evidence standards to accommodate the new advances in digital technologies, such as AI- and ML-based innovations, can be accomplished within the context of HTA policy sandbox programme to facilitate early and continuous engagement with these technologies’ developers and other stakeholders with the aim of mutual learning and adaptation. Closer working with stakeholders including regulators, digital health developers, clinicians, patients, academics and guideline developers will facilitate an iterative process of product development and ensure faster access to and uptake of innovation in the healthcare system [4].

The appraisal of other innovative technologies such as advanced therapy medicinal products including gene and cell therapies and histology-independent cancer drugs also represent a challenge to HTA agencies given the lack of experience in appraising these technologies [40]. Horizon Scanning and Scientific Advice activities implemented by some HTA agencies have been useful in this context [2, 41]. However, these do not go a long way towards achieving this anticipatory environment that allow development of HTA policy and methods in response to new innovations. Using an anticipatory sandbox approach to facilitate the development of appraisal methods, policies, and processes and to adapt to the unique features of these disruptive innovations would enable HTA agencies and all stakeholders involved to have a constructive discussion that would facilitate faster uptake and patient access to these technologies while supporting innovation and developing HTA methods, processes and policies.

Thus, based on the findings of this review we recommend the following as the key elements to focus on when developing a HTA policy sandbox (Fig. 5):

Fig. 5

Recommendations for developing HTA policy sandbox

Focusing on HTA policy and methods: there is limited activity within this area, as many sandboxes currently focus on regulatory issues only, and do not include HTA within the product development pathway. A HTA sandbox would therefore complement already established regulatory sandbox programmes.

Focus on future advances and innovations in healthcare that do not fit within an established appraisal pathway: there are many products that do not fit within current processes that have the potential to provide a large population benefit, particularly those making use of AI and ML approaches.

Using an anticipatory approach, as it is important that HTA prepares for future innovations rather than being reactive. Through an anticipatory approach, it is possible for HTA agencies to directly influence innovation, minimise risks, optimise outcomes and prevent population harm.

Using consortium working, to ensure successful collaboration between multidisciplinary stakeholders, creating outputs that are transferable across a variety of settings.

Our review supports establishing such sandboxes by HTA agencies, however, research is needed to understand how to embed these sandboxes in the organisational structure of current HTA agencies, managing the changes that accompany their implementation, and evaluate their benefits. It has to be pointed out as well that the examples identified in this review have been mainly from high-income countries, and most from the UK. This might be a result of language limitation. However, we have identified literature on regulatory sandboxes in sectors other than healthcare in the newer EU countries (the EU13) [42]. Despite previously being identified as having an ‘innovation gap’ compared to other older member states, some countries have chosen to adopt innovation hubs which are similar to regulatory sandboxes in their aims of providing regulatory guidance [43]. This supports the transferability of this concept to HTA agencies in low- and middle-income countries as well.

Conclusion

The sandbox approach has been used across a variety of sectors since its implementation in the finance sector. Healthcare regulatory agencies have developed sandboxes focused on the regulation of new healthcare innovations, primarily those related to digital health with successful outputs reported. HTA sandboxes could be useful in anticipating the challenges in assessing innovative technologies and future-proofing methods and processes.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 44 kb)

The sandbox approach originated in the financial technologies sector and since then has started to be used within healthcare for developing regulatory policies relating to innovative technologies.

This approach has considerable potential to be used in accelerating health technology assessment (HTA) methods and policy developments, particularly in response to new innovations.

The success of applying this approach in HTA should be facilitated by consortium working, clarity regarding scope, focus on innovative technologies with no clear appraisal pathway and the use of an anticipatory model.