The relationship between the living lab approach and successful implementation of healthcare innovations: an integrative review.

OBJECTIVES: The concept of living labs as a research method to enhance participation of end-users in the development and implementation process of an innovation, gained increasing attention over the past decade. A living lab can be characterised by five key components: user-centric, cocreation, real-life context, test innovation and open innovation. The purpose of this integrative literature review was to summarise the literature on the relationship between the living lab approach and successful implementation of healthcare innovations.
METHODS: An integrative literature review searching PubMed, EMBASE, PsycINFO and Cinahl databases between January 2000 and December 2019. Studies were included when a living lab approach was used to implement innovations in healthcare and implementation outcomes were reported. Included studies evaluated at least one of the following implementation outcomes: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration or sustainability. Quality was assessed based on a tool developed by Hawker et al.
RESULTS: Of the 1173 retrieved articles, 30 studies were included of which 11 of high quality. Most studies involved a combination of patients/public (N=23) and providers (N=17) as key stakeholders in the living lab approach. Living lab components were mostly applied in the development phase of innovations (N=21). The majority of studies reported on achievement of acceptability (N=22) and feasibility (N=17) in terms of implementation outcomes. A broader spectrum of implementation outcomes was only evaluated in one study. We found that in particular six success factors were mentioned for the added-value of using living lab components for healthcare innovations: leadership, involvement, timing, openness, organisational support and ownership.
CONCLUSIONS: The living lab approach showed to contribute to successful implementation outcomes. This integrative review suggests that using a living lab approach fosters collaboration and participation in the development and implementation of new healthcare innovations. PROSPERO REGISTRATION NUMBER: CRD42020166895. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Summarises the literature on the relationship between the living lab approach and successful implementation of healthcare innovations based on the implementation outcomes suggested by Proctor et al.

Includes a broad search terms in order to understand what components of the living lab approach are currently applied.

Studies were included irrespective of study design (integrative review) and successful implementation was evaluated even when only one implementation outcome was reported.

As a shortcoming is the use of the Hawker et al quality appraisal tool as the tool itself does not suggest cut-off values for the overall quality assessment.

Most studies only evaluate acceptability and feasibility in terms of implementation and not the full range of suggested implementation outcomes by Proctor et al.

Background

The concept of living labs as a research method to enhance participation of end-users in the development and implementation process of an innovation gained increasing attention over the past decade.1 In Europe, the application of living labs in real-life settings and ‘real’ experimentation emerged around 2005. In line with strengthening democratic processes in the EU, policies strongly encourage collaborative approaches in order to create innovation and the involvement of stakeholders by including them into the design and implementation of different fields of research and development.2 At first, living labs mostly emerged from Information and communication technology and urban developments to test innovations in a real-world environment.3–5 In 2006, the European Network of Living Labs (ENoLL) was formed as an international collaboration platform. The living lab approach has, in the following years, also been adopted in healthcare settings with the introduction of several European living labs.6–10 Programmes, such as Horizon 2020, promote the use of the living lab approach, including its application in the field of healthcare and health promotion. In 2018, more than 440 living labs have been recognised in Europe. ENoLL identified 69 of the current living labs as being health-related concerning diverse topics such as ageing, healthy living and mobility, chronic diseases and technological innovation.11 Currently, no consistent or commonly accepted definition of living labs exists, but the following terms are considered key components of living labs: user-centric, cocreation, real-life context, test innovation and open innovation.1 3 12–15 The goal of living labs is to develop useful and usable products and/or services to create value.1 15 Also, in terms of the current discussion on political agendas concerning the involvement of the public in innovations, living labs offer the possibility to engage the public in the process of innovation development and implementation.16 For the purpose of this integrative review, a living lab is defined as a user-centric research methodology for developing, testing and implementing complex healthcare innovations in a real-life context. An example of a Dutch living lab is the eLabEL project which aimed to improve integrated digital support in primary care. Stakeholders consisting of patients, healthcare professionals, entrepreneurs and researchers collaborated during the selection, integration, implementation and evaluation of developed eHealth-tools in primary healthcare.17 In the living lab, stakeholders together identified needs and expectations of eHealth solutions followed by several sessions to integrate the chosen eHealth solutions.17 In this sense, complex innovations include ideas, practices or technologies that are new to the end-user and that require the involvement of multiple stakeholders to use and implement to achieve better quality of care.18

A recent literature review explored the concept of living labs to investigate population specific health-related problems and the application of five common elements of living labs, namely multimethod approach, user engagement, multiple stakeholders, real-life settings and cocreation.19 The authors found that all of the five key elements were used in most studies. Moreover, this review suggests that using a living lab approach helps to improve physical, social and cognitive health. However, a living lab approach does not exclusively concern the developmental process of products and/or services, but also ensures sustainable implementation.20

Implementation of research findings are essential to enhance timely adoption to improve quality of care.21 It is estimated that approximately two-thirds of efforts to implement change are not successful.22 Possible barriers include, for example, awareness, motivation to change, attitude and involvement on an individual professionals’ and patients’ level.23 Implementation concerns a set of purposeful processes and/or activities specifically developed to put an intervention or programme into practice.24 In order to assess successful implementation, measures need to be used that are distinct from those that assess effectiveness of an intervention. This distinction is crucial, as success or failure of innovation can be due to, for instance, an ineffective intervention or insufficient reach and/or incorrect use in practice. The incorrect use in practice and insufficient reach concerns assessment of successful implementation. Barriers to successful implementation include insufficient involvement and support, poor dissemination strategies and lack of leadership and willingness to change.25–27 Outcome measures for implementation have been proposed by Proctor et al and include the following: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration and sustainability.28 Successful implementation is proposed to be measured as an equation of the effectiveness of an intervention being implemented plus the described implementation factors.28 For the purpose of this integrative literature review, successful implementation will be measured as proposed by Proctor et al based on the implementation outcomes.28 The living lab approach may support successful implementation, as end-users are not only involved in the development but also testing of the innovative products. Therefore, the goal of this integrative literature review is to assess the literature on the relation between the living lab approach and successful implementation of innovations. By doing so, the aim is to (1) identify which key components of the living lab approach were used; (2) identify which implementation components were measured; and (3) determine what the relationship is between the living lab approach and successful implementation of innovations.

Methods

This integrative review aimed at including all available literature in the field to draw an understanding of the relation between the living lab approach and successful implementation of innovations. Methods of an integrative review allow for the inclusion of different study designs (qualitative and quantitative).29 Results of the review are reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (online supplemental file 1).30

Information sources and literature search

To identify relevant publications, a systematic search was performed in the bibliographic databases PubMed, Embase, Cinahl and PsycINFO from January 2000 to December 2019. Additionally, snowball strategies were used to screen reference lists of eligible papers. Search terms included free-text terms to capture the concept of “living lab” (eg, ‘co-creation’ or ‘co-design’) and “successful implementation” (eg, ‘fidelity’ or ‘implementation evaluation’). The concept of “successful implementation” entails the evaluation of at least one of the Proctor et al implementation outcomes. An information specialist was consulted in the development phase of the search strategy. The full search strategy tailored for all databases can be found in online supplemental file 2.

Study selection

The goal was to include studies that used a living lab approach in either of the following phases of an innovation: development, implementation or evaluation. Studies that report on a minimum of one implementation outcome were included in this study. For the purpose of this integrative review, implementation was defined as purposeful activities designed to put a programme or activity into practice.24 Studies evaluating or assessing at least one or more of the following implementation outcomes as proposed by Proctor et al were eligible for inclusion to evaluate successful implementation: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, sustainability.28 The implementation outcomes are described in table 1. In order to determine the relation between the living lab approach and implementation, studies reporting on success factors for the implementation due to the application of the living lab approach were included. Moreover, all studies were included irrespective of study design. Only full-text articles published in English, German or Dutch were included. The search was restricted to these languages as this covered the expertise of the research team. Studies not concerning living lab approaches in healthcare were excluded, as well as concept papers describing the methodology of living labs without evaluation of implementation. Additionally, commentaries, editorials, letters and books were excluded. Grey literature including conference abstracts and dissertations were also not included as the goal was to assess peer-reviewed literature to explore the relation between the living lab approach and successful implementation. First, two reviewers (NZ and BH) independently checked all retrieved titles and abstracts. Second, full-text articles were screened and selected. Additionally, through the backward snowball method, reference lists of selected articles were checked for possible relevant studies.31 Three papers were assessed through backward snowballing.32–34 Disagreement between the two reviewers (NZ and BH) was resolved until consensus was reached with the help, if needed, of a third reviewer (SJvdB-V).

Table 1

Description of implementation outcomes adapted from Proctor et al28

Implementation outcome	Definition	Key aspect	Stage during implementation process
Acceptability	The perception among implementation stakeholders that an innovation is agreeable, palatable or satisfactory	Based on direct experience of stakeholders	Early, mid and late stage
Adoption	The intention, initial decision or action to employ an innovation.	Based on the perspective of the provider or organisation	Early and mid-stage
Appropriateness	The perceived fit, relevance or compatibility of the innovation for a given setting and fit to address a particular problem.	Based on the perception from involved stakeholders but also organisation	Early stage
Feasibility	The extent to which an innovation can be successfully used or carried out within a particular setting.	Mostly assessed retrospectively	Early and mid-stage
Fidelity	The degree to which an intervention was implemented as prescribed in the original protocol or as intended.	Includes adherence, quality of delivery, programme component differentiation, exposure to intervention and participant involvement	Early and mid-stage
Implementation cost	The cost impact of an implementation effort.	Costs concerning delivery, the innovation itself, the implementation strategy and location for the service delivery	Early, mid and late stage
Penetration	The integration of a practice within a service setting and its subsystems.	Based on the number of providers who delivered the innovation or the reach of the innovation	Mid-stage and late stage
Sustainability	The extent to which a newly implemented innovation is maintained.	Includes permanent funding and integration in routine on individual and organisation level	Late stage

Extraction of data and analysis

Data extraction was performed to identify which key components of the living lab approach were used and which implementation outcomes were measured in relevant studies. The following information was collected: first author, year, country, target population for innovation, innovation type, aim of the innovation, study design (with detailed explanation), stakeholder type for cocreation or codevelopment of the innovation, programme design and characteristics of the cocreation, purpose of the cocreation programme, outcome of the cocreation programme (process outcomes), living lab key components (user-centric, cocreation, real-life context, test innovation, open innovation), implementation aspects (acceptability, adoption, appropriateness, feasibility, implementation cost, penetration, sustainability), implementation outcome, and relationship between cocreation programme outcome and implementation outcome to determine the relationship between the living lab approach and the implementation. For the implementation aspects the outcomes as recommended by Proctor et al were used.28 Data extraction was divided among two reviewers (NZ and BH) and checked vice versa by the other reviewer. Disagreements were discussed and resolved until consensus was reached. In case of a study referring to another publication for further description of the design or other relevant information, the additional publication was used to add to the data extraction. Data were synthesised through narrative synthesis due to the diverse study designs included and used the outcomes by Proctor et al as a synthesis taxonomy.28 35 36 The goal was to explore the relationship between the cocreation programme outcome and implementation outcome. Textual descriptions were conducted for each included study based on the predefined data extraction sheet. The taxonomy by Proctor et al was used to classify previous research.28 35 After tabulation of data, aspects of each study were textually described.37 Due to the heterogeneity of studies, a pooled effect was not assessed.

Quality assessment

A quality assessment was performed to score the quality of the included studies in terms of methodological rigour of studies based on the tool from Hawker et al38 as it was deemed most appropriate for the heterogeneous articles included in this systematic review (online supplemental file 3). The quality assessment tool was chosen as it covers a variety of research paradigms, which was specifically suitable as we did not discriminate based on study design, but wanted to get a broad picture of different study approaches. Moreover, the quality assessment tool by Hawker et al offered a clear description of the scoring for the following nine categories: abstract and title; introduction and aims; method and data; sampling; data analysis; ethics and bias; findings/results; transferability and generalisability, and implications and usefulness with a maximum score of 36 in total (also see online supplemental file 2). Methodological quality was assessed for each item (4=good; 3=fair; 2=poor; 1=very poor). The quality appraisal by Hawker et al does not propose cut-offs for the quality assessment. Therefore, methodological quality was determined based on the earlier suggested cut-offs by Braithwaite et al. ‘high quality’ (30–36 points), ‘medium quality’ (24–29 points) and ‘low quality’ (9–23 points).39 After applying the Hawker tool to the studies, the categories ‘good, fair, poor and very poor’ were converted into a numerical score by assigning the answers from 1 point (very poor) to 4 points (good). Then, a score was produced for each study with a minimum of 9 points and a maximum of 36 points. The following definitions were used to create the overall quality grades: high quality (30–36 points), medium quality (24–29 points) and low quality (9–24 points). Two reviewers (NZ and BH) performed the quality assessment independently and disagreements were resolved until consensus was reached.

Patient and public involvement

No patients involved.

Results

The final systematic search resulted in N=1173 unique articles for initial screening; N=171 were included for full-text screening of which N=141 were excluded as they did not meet the inclusion criteria. Figure 1 shows the results of the screening process according to the PRISMA diagram and reasons for exclusion. In total N=30 studies were included for data synthesis.

Figure 1

PRISMA flow diagram of study inclusion. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Study characteristics

Table 2 presents the characteristics, living lab components, phase of innovation and implementation outcomes of included studies (N=30). Studies were conducted in 12 different countries. Most studies were conducted in the USA (N=7),40–46 the UK (N=6)47–52 or Canada (N=5).53–57 The majority of studies applied a mixed-methods study design (N=10)42–46 51 53 55 56 58 or a qualitative study design (N=6).48 52 54 57 59 60 For eliciting the stakeholder type, categorisation to sum up the stakeholder types was used.61 According to the so-called 7Ps Framework to identify stakeholders in patient-centred outcomes research, the following stakeholder groups are of interest: patients and the public, providers, purchasers, payers, policy-makers, product makers and principal investigators.61 Most studies involved a combination of patients and public (N=23)40–46 48–55 57–60 62–65 together with providers (N=17)42 44 47–49 51–58 60 63 66 67 in the living lab approach. The combination of patients and public together with providers was used in N=10 of the included studies. Notably, only one study involved policy-makers and none of the included studies involved purchasers, payers or product makers.68 The most common living lab key components identified were cocreation (N=25)40–42 44–51 53 55–60 62–68 and user-centric (N=10).40 41 43 46 48 52 54 60 66 67 A combination of the living lab components cocreation and user-centric were mentioned in N=5 studies.40 46 48 60 67 Three key components of the living lab approach were mentioned in N=5 studies.41 46 52 62 66 The living lab key component open innovation was not mentioned in included studies.

Table 2

Characteristics of included studies (N=30)

Characteristic	No of studies	References
Country
Australia	N=4	62–64 66
Canada	N=5	53–57
China	N=1	65
Ethiopia	N=1	68
Mexico	N=1	69
The Netherlands	N=1	48
New Zealand	N=1	58
Northern Ireland	N=1	59
Portugal	N=1	60
UK	N=6	47–52
USA	N=7	40–46
Study design
Qualitative study	N=6	48 52 54 57 59 60
Mixed-method study	N=10	42–46 51 53 55 56 58
Process evaluation	N=3	47 49 68
(Quasi)-experimental study design	N=9	40 41 50 62–67
Type of stakeholder in the living lab approach
Patients and the public	N=23	40–46 48–55 57–60 62–65
Providers	N=17	42 44 47–49 51–58 60 63 66 67
Purchasers	N=0	N/A*
Payers	N=0	N/A
Policy makers	N=1	68
Product makers	N=0	N/A
Principal investigators	N=4	49 57–59
Living lab component
User-centric	N=10	40 41 43 46 48 52 54 60 66 67
Cocreation	N=25	40–42 44–51 53 55–60 62–68
Real-life context	N=8	41 46 47 52–54 62 68
Test innovation	N=4	46 52 62 66
Open innovation	N=0	N/A
Phase of innovation
Development	N=21	40–42 44 47–49 51 53 55 57–60 62–68
Implementation	N=11	43 45 49 50 52 54 56–60
Evaluation	N=10	42–45 48–50 54–56
No of cocreation steps in the living lab
Two cocreation steps	N=8	42 51 54 55 58 62 68 69
Three cocreation steps	N=5	46 56 57 59 60
Five cocreation steps	N=4	41 49 64 66
None stated	N=12	40 43–45 47 48 50 52 53 63 65 67
Implementation outcome†
Acceptability	N=22	42–44 46 47 49–55 57–59 62–68
Adoption	N=5	40 47 48 60 68
Appropriateness	N=1	51
Feasibility	N=17	42–46 49 50 52–55 58 59 63 65–67
Fidelity	N=4	40 45 56 68
Implementation cost	N=2	40 56
Penetration	N=1	40
Sustainability	N=4	40 49 57 64

*N/A means not available.

†For the implementation outcome it was possible to have multiple outcomes reported for one study.

Living lab components were mostly applied in the development phase of interventions (N=21).40–42 44 47–49 51 53 55 57–60 62–68 The most mentioned combination was development and implementation phase (N=5)49 57–60 and development and evaluation phase (N=5).42 44 48 49 55 In the studies using cocreation in their living lab approach, studies reported on different stages for their cocreation process (n=17).41 42 46 49 51 54–60 62 64 66 68 69 These stages include, for instance, exploration, ideation and reflection and implementation60 or prototyping and testing.68 The minority of studies using the living lab approach did not discriminate different stages (N=12).40 43–45 47 48 50 52 53 63 65 67 Studies that mentioned cocreation as a living lab component, but did not describe cocreation phases, reported on, for example, a cyclical process with various meetings.48 Two studies made use of formal frameworks for their living lab approach including the Behaviour Change Wheel66 and/or the Theoretical Domains Framework.63 The use of these frameworks shaped the analysis of the qualitative results. For the implementation outcomes, it was possible that more than one outcome was studied and reported on in a single study. The most reported implementation outcome was acceptability (N=22),42–44 46 47 49–55 57–59 62–68 which was often combined with feasibility (N=17).42–46 49 50 52–55 58 59 63 65–67 Remarkably, only one study reported on appropriateness (N=1)51 and one on penetration (N=1).40

Included studies were classified as either high, medium or low quality. The quality scores ranged from 21 to 35 across the 30 included articles. The results of the quality assessment are presented in table 3. Of the included studies, 12 were considered of high quality.47 48 50 52 53 57 62 63 65–67 Three of the included studies scored low on the quality assessment.44 49 69 The detailed results of the quality assessment can be found in online supplemental file 2.

Table 3

Methodological rigour and quality of included articles

Quality classification*	Points scored on the Hawker et al quality assessment tool*	No of articles classified in each section	References
High	30–36	11	47 48 50 52 53 57 62 63 65–67
Medium	24–29	14	40–42 45 46 51 54–56 58–60 64 68
Low	9–23	3	44 49 69

*Adapted from cut-off values determined by Lorenc et al.83

Results

Overview of reported implementation outcomes

In terms of implementation success, most of the studies (N=20/30) reported on a positive implementation outcome40 44–48 50 51 54–56 58 60 62–65 67–69 (see table 4). The results indicated mostly successful implementation in terms of acceptability and feasibility of a healthcare programme or intervention development.44 46 50 51 54 55 58 62 63 65 67 The desired implementation outcomes were not achieved in all studies. For example, one study reported on a positive feasibility outcome, but not on acceptability.53 Although considered feasible the programme did completely transfer the learning into practice and did not lead to significant changes in service delivery compared with before.53 This study, however, still achieved a high quality score according to the quality assessment tool.53 Another study did not describe results regarding acceptability and feasibility but reported on the sustainability of the intervention.49 In this study, sustainability was secured by providing future funding, contracting, a protocol to regularly update the content of the innovation and through facilitating wider spread of the innovation.49 Furthermore, N=3 studies did not achieve their implementation outcome at all.52 57 59 Two studies reported that the intervention was not found feasible.52 59 In one of these, participants in the living lab approach indicated that it was not feasible to apply the living lab approach when also being in charge of resourcing for the execution of the living lab approach.59 The other study still needs to evaluate the feasibility result of the intervention.52 One of the included studies evaluated five of the proposed implementation outcomes.40 They reported that the intervention was more successful in one of the collaborative teams in comparison with the other usual care teams. The characteristics of the successful team are in line with the success factors for the implementation as that team involved middle and senior managers, whereas the other team only involved frontline end-users.40 Moreover, the successful team had a a priori set focus for change, whereas the other team had less of a strict strategy.

Table 4

(A) Synthesis of results in terms of implementation outcomes. (B) Continued synthesis of results in terms of implementation outcomes. (C) Continued synthesis of results in terms of implementation outcomes

Author/year	Hawker et al appraisal low/medium/high	Implementation outcome(s)	Implementation outcome achieved +*/-†/0‡	Result of implementation and relation to the living lab approach
Li,65 2019	High	Acceptability	+	The intervention was highly acceptable by the participants based on the high overall attendance and satisfaction for the content and mean agreement. Acceptability was enhanced by the patient participatory approach for optimising the intervention content.
		Feasibility	+	Pilot data showed feasibility with regard to content, length of the programme and programme objectives.
Morgan,57 2019	High	Acceptability	–	Higher acceptability due to inclusion of the first link coordinator in the clinic assessment.
		Sustainability	–	Participants stated continuing the connection with the research team, involvement and informing all managers, having consistent leaders to ensure ongoing support, and increasing the community awareness of the service as contributing to sustainability of the intervention.
Timmerman,48 2016	High	Adoption	–	For future adoption and implementation willingness in using a telehealthcare system and using a single platform that integrates all functionalities is important. Healthcare professionals (HCPs) reported usability issues. Involvement of end-users is reported to be essential to contribute to adoption, compliance and implementation.
Bolton,50 2016	High	Acceptability	+	Intervention was acceptable to implement. Participants were able to make changes to the plan to suit their needs. Result of the Social Support Programme Acceptability Rating Scale were between the categories 2=quite a lot and 3=a great deal.Community engagement enhanced acceptability to the participants.
		Feasibility	+	It was feasible to devise and implement a community-led, community-level intervention providing social support to new mothers.
Tsianakas,67 2015	High	Acceptability	+	Intervention was acceptable to both carers and HCPs, they reported the intervention was educative and increased confidence.
		Feasibility	+	The recruitment rate of ten carers per month and low attrition proved feasible in the study setting.
Gould,63 2019	High	Acceptability	+	Evidence showed that the content education and training was acceptable to a wide range of the staff. 55% of women accepted (Nicotine replacement therapy). Taking into account cultural and other context aspects, and specific needs of the community contributed to the acceptability.
		Feasibility	+	Feasibility was shown by moderate to high retention (77% and 40%) and recruitment rates (47% and 54%).
Engelen,62 2019	High	Acceptability	+	Participants (94% of respondents) found the intervention informative and useful in conveying Move More Sit Less messages in an easily accessible format. The codesign assured culturally acceptable components and higher acceptability.
Tatla,53 2017	High	Acceptability	–	There were no significant changes in service delivery pre- and post-intervention (p<0.05).
		Feasibility	+	The programme was feasible to deliver and for the measurement of outcomes (processes of care and coaching skills or interprofessional learning outcomes).
Bonner,66 2019	High	Acceptability	+	Overall acceptability was rated 8.4/10. 88% intended to use the website over the next month suggesting acceptability. The user codesign of the intervention also made the intervention acceptable to the end-users (GPs).
		Feasibility	+	At the 1-month follow-up most GPs (73%) reported using the website.
Williams,47 2016	High	Fidelity	+	Environmental health practitioners delivered the risk audit as prescribed and it was easy to use.
		Adoption	+	The coproduction of the intervention proved important in facilitating adoption.
		Acceptability	+	The coproduction approach produced an acceptable intervention.
Zamir,52 2018	High	Acceptability	+/-	Sites where the intervention was accepted, embodied an activity led environment and staff were accustomed to dedicating their time to engage in activities.
		Feasibility	0	Feasibility is yet to be determined by making observations and evaluating outcomes.
Magge,68 2019	Medium	Acceptability	+	Acceptability was shown by the willingness of the federal ministry of health to dedicate employees full time to the initiative.
		Adoption	+	35 quality improvement teams have been actively engaged with 83 change ideas demonstrating high levels of adoption. Aligning the collaborative design with the district-wide system was seen as instrumental for the integration into existing administrative structures.
		Fidelity	+	The collaboratives were completed and implemented as designed with few adaptations.
Cameron,64 2019	Medium	Acceptability	+	Acceptability increased by tailoring the tools and recourses to specific needs of the site.
		Sustainability	+	Sustainability increased by tailoring the tools and recourses to specific needs of the site. Sustainability was ensured by broadening the intervention and incorporating it into existing processes.
Shah,582019	Medium	Acceptability	+	It was stated that the intervention was well accepted by the patients. The codesign approach enabled clinical staff to have input into the content, the implementation and the resource material used. Involving different cultures, and the translation made the research culturally appropriate and likely increased participation.
		Feasibility	+	Responses to the feasibility of the intervention were mainly positive. GPs reported that it enabled patients to think about issues before their consultations.
Easton,51 2019	Medium	Acceptability	+	Results showed it was acceptable to receive support for self-management and acute exacerbations from an AI-based virtual agent. Codesigning the content and application of the virtual agent made the system acceptable to the target population.
Grenha Teixeira,602019	Medium	Adoption	+	User adoption numbers showed the intervention was well received by professionals, with more than 1.8 million citizens and 650 institutions that were registered in the Portuguese HER and an average of 100 000 healthcare practitioners daily accesses and 12 000 citizens accesses to the healthcare information. The participatory approach was used to address user adoption challenges.
Kipping,56 2019	Medium	Fidelity	+	Fidelity checks showed that units were converging to an average of 78%. Results suggest that the cocreation process is an effective implementation strategy for model fidelity. Staff and patient involvement in the design and creation of new practices within their local environment led to greater adherence to fidelity.
		Implementation cost	+	The implementation cost was US$80 974.14.
Whitehouse,552013	Medium	Acceptability	+	100% of the professionals stated that the platform had an acceptable format and the use was easy. Youth acceptance of using the tool in the study was exceptional, demonstrating the importance of being responsive to user suggestions from inception.
		Feasibility	+	The professionals found the tool useful and it met their needs, uptake was 99%.
Tolma,452019	Medium	Feasibility	+	All of the eligible research participants met with the medical provider and 80% participated in the discussion groups.Using multiple level intervention targets at clinic and community settings through community-based participatory research (CBPR) are feasible.
		Fidelity	+	Most of the activities, including doctor-patient guided communication, breast cancer brochure, bulletin board poster and mammogram screening flowchart of the clinic-based component took place as planned except the physician’s recommendation letter. Most of the community components, the intergenerational group discussion and bracelets as a reward for getting a mammogram, took place as planned.
McConnell,59 2018	Medium	Acceptability	0	No results
		Feasibility	–	Participants reported that it may not be feasible to take on the living lab approach when they also have to be responsible for resourcing and capacity building for achieving long-term goals.
Crosby,42 2017	Medium	Acceptability	+	The iManage app prototype was rated as easy to use, beneficial for tracking sickle cell disease and participants liked that the app was tailored to their needs.
		Feasibility	+	The participants accessed the internet through devices and reported seeking information about their condition online (71%), which provides initial support for the feasibility of mobile health intervention.
Voyer,54 2014	Medium	Acceptability	+	Results suggest that the programme is acceptable to healthcare staff because of the internal support, effective clinician leadership, taking into account the internal culture and policies, fostering a sense of ownership and providing practical training in addition to the theory.
		Feasibility	+	Results suggest that the programme is feasible because of the internal support, the effective clinician leadership, taking into account the facility’s own internal culture and policies, fostering a sense of ownership among the users and providing practical training in addition to the theory.
Berge,41 2016	Medium	Feasibility	+	Over half of the families in the intervention group attended 75% of the events and 33% attended all events. Using local parks and elementary schools was highly feasible, also using social media to advertise the intervention was highly feasible.
		Sustainability	0	Not evaluated in results
Dugan,40 2016	Medium	Adoption	+	For the Kaizen Event Team interventions one was successful in terms of adoption two were unsuccessful and one is in progress For the Design Team interventions one was successful in terms of adoption, 1 did not occur and one is in progress
		Fidelity	+	For the Kaizen Event Team interventions one was successful in terms of fidelity, 1 was unsuccessful, 1 was not available and one is in progress. For the Design Team interventions one was successful in terms of fidelity, 2 are in progress and one did not occur.
		Implementation cost	+	For the Kaizen Event Team interventions one was successful in terms of cost effectiveness, 1 was unsuccessful, 1 is not available and one is in progress. For the Design Team interventions one was not available, 1 needs to be determined, 1 is in progress and one did not occur in terms of cost effectiveness
		Penetration	+	For the Kaizen Event Team interventions one was successful in terms of penetration, 1 was unsuccessful, 1 was not available and one is in progress and needs to be determined. For the Design Team interventions one was not available, 2 are in progress and need to be determined and one was successful in terms of penetration
		Sustainability	+	For the Kaizen Event Team interventions one was successful in terms of sustainability, 1 was unsuccessful, 1 was not available and one is in progress and needs to be determined. For the Design Team interventions one was not available, 2 are in progress and need to be determined and one was successful.
Black,46 2018	Medium	Acceptability	+	Staff reported the video’s and online programme were helpful and easy to administer.
		Feasibility	+	Youth were able to complete the seven chapters rooted in (American Indian and Alaska Native) beliefs of the intervention with little assistance and the online format provided flexibility that accommodated inconsistent attendance. Contributions of two community partners groups in the implementation ensured that the intervention was compatible and feasible.
Chávez,69 2019	Low	Fidelity	+	A high fidelity prototype was developed.
Prochaska,44 2011	Low	Acceptability	+	Participants responded positive to the acceptability measures and strong use of computers made the computer-delivered intervention an acceptable way to implement the programme. Collaborative involvement of key stakeholders can enable the continued development of an acceptable and feasible programme, enabling thereby the programme of research to be translated to practice.
		Feasibility	+	Ease and speed of the recruitment with exceeding the goal of 50 women to 87 in 3 weeks, the ability of participants to complete the programme within 20–30 min and high participation rate (all women agreed to participate) showed the feasibility of incorporating the programme at health centres.
Horne,49 2016	Low	Acceptability	0	Results of evaluation of acceptability not shown.
		Feasibility	0	Results not shown.
		Sustainability	+	Sustainability was ensured by secured future funding, contracting, a protocol to update the website content, and clinical facilitation of the forum and the messaging system. The website was tailored and developed by a multidisciplinary team and sustainable.

*+ means that the implementation outcome was successful.

†- means that the implementation outcome was not successful.

‡0 means that the implementation outcome was not reported or evaluated.

The relationship between the living lab approach and implementation

The living lab approach, due to its participatory nature, showed to lead to successful implementation outcomes. The included studies that applied the living lab approach reported on six success factors for the implementation:

Leadership: Leadership in the collaborative and participatory approach of a living lab was seen as crucial to the success of the implementation.44 45 53 54 57 62 68 The presence of senior leadership might contribute to better involvement of end-users53 as well as participants that are already familiar with the intervention to support the implementation.62 In contrast, resistance of senior leaders can hinder the implementation.55

Involvement: Studies made use of participatory techniques to involve end-users in all phases of the living lab. Involvement of end-users was found as a catalyst for the implementation.40 44 48 54 56 57 59 62 68 69 The involvement of end-users early on in the process of a healthcare programme or intervention development can contribute even further to successful implementation outcomes.51 59

Timing: The timing and continuity of participatory living labs may enhance the success of the implementation.40 Unrealistic timeframes for speedy implementations of healthcare innovations can hinder successful implementation as the living lab approach with its participatory nature requires time. However, long durations of implementations can also hinder motivation of participants in the living lab.

Openness: A factor that might contribute to better implementation in a living lab approach, as reported, was openness for change.53 The commitment and willingness of participants in a living lab can support better implementation.53 Openness can also be linked to cultural aspects within organisations54 as well as open communication.64

Organisational support: Many of the studies included a combination of patient and professional involvement. In order for professionals to be able to commit to a participatory living lab, organisational support was found as a success factor.53 54 63 Support from the organisation also includes capacity to support a living lab in terms of staff as well as funding.45 49 58 63

Ownership: A sense of ownership of the healthcare programme or intervention to be implemented with the living lab approach might contribute to more successful implementation outcomes.54 This also includes meeting the needs of end-users for the programme to be seen of added-value42 and shared responsibility.59

These success factors, as identified from the included studies, may contribute to the achievement of desired implementation outcomes when applying the living lab approach.

Discussion

The purpose of this integrative review was to summarise the literature on the relationship between the living lab approach and successful implementation of healthcare innovations. The results of this review show that cocreation and user-centric were the most applied living lab key components in the included studies. Most studies reported achievement of the desired implementation outcomes. However, most studies only evaluated one or two implementation outcomes. Six success factors for implementation due to the living lab approach were identified: leadership, involvement, timing, openness, organisation support and ownership.

Summary of evidence

Most of the included studies made use of a participatory or collaborative design for their living lab approach. The terms participation or collaboration were mostly used interchangeably while implying the same construct of involving several actors in the process of developing, implementing and evaluating an innovation. In the literature, collaboration is defined as ‘the possibility to gather active contribution from several actors during a creative process’.70 Participation entails the possibility to intervene in the development of an innovation by users in order to meet their needs.70 Living labs are increasingly emerging as they promise to meet the public and policy interest for developing and implementing innovations in collaboration with the public.16 However, the lack of a clear definition of a living lab makes it challenging to determine whether the achievement of two or fewer living lab components in included studies make them living labs as per definition. The proposed definitions in the literature are rather narrative instead of offering clear characteristics that need to be met in a collaborative design to act as a living lab.71 It is, therefore, arguable whether the included studies should be evaluated as full living labs when they were only considering two or fewer living lab components. However, living labs offer the unique possibility to prevent the issues concerning implementation and uptake of innovations in healthcare due to limited public trust or clinical resistance, for example.72 In our review, specifically user-centric and cocreation were the most common components. This result is in line with the proposed definition of a living lab: ‘A living lab is a design research methodology aimed at cocreating innovation through the involvement of aware users in a real-life setting.’71 In our study, we did not elicit whether the number of living lab components used can be linked to the level of success in implementation outcomes. However, we identified six success factors for achieving implementation. These identified success factors are not an exclusive list, but an overview of the most prominent success factors also identified in earlier studies.9 Leadership was mentioned as a facilitator for successful implementation in a collaborative design approach.44 45 53 54 57 62 68 Specifically, senior leadership was identified as an important prerequisite. Besides senior leaders to support the new innovation, however, intermediate or lower level leadership support needs to be present for successful implementation on a larger scale.73 The collaborative character of living labs require committed involvement of all actors, which is also supported in the literature.74 In particular, the involvement of patients who are ultimately the ones benefitting from new innovations in healthcare can help to meet their needs.75 Our study also found that mostly patients and the public were involved in the cocreation process which is in line with an earlier integrative review on living labs.19 In order to achieve involvement, an open culture on an organisational, but also individual level, is required,53 54 whereby the composition of actors needs to be considered.76 The identified success factors may contribute to more favourable implementation success, but causal relations between the success factors and an implementation outcome as suggested by Proctor et al cannot be made. In an earlier systematic review success factors for the implementation of open innovation, which is a component of a living lab, were identified.77 Their results are partly in line with our results including findings as the importance of leadership, network and relationship which is linked to involvement and openness, and culture which is linked to our finding of organisational support and openness.77 Our integrative review on the relation between the living lab approach and implementation success contributes to the existing body of evidence as it identified gaps in current research and bridges knowledge between the fields of implementation science and literature on aspects of living labs.

In terms of implementation outcomes, the most commonly evaluated outcomes were acceptability and feasibility. Of the included studies, N=13 measured acceptability and feasibility together. Most of those studies were of high quality in terms of Hawker scoring. One study achieved the desired implementation outcome, but was of low quality.44 Even though, most studies reported on achieved implementation outcomes, most studies only gave insight into few of the Proctor outcomes and it remains unknown how the implementation scored in terms of the other Proctor outcomes. An earlier systematic review also found that acceptability is the most used outcome as it has a long history in theoretical as well as in the empirical literature.78 For feasibility, measurement instruments are however scarce, which would also explain the rather narrative results found in our review.78 The list of implementation outcomes as proposed by Proctor et al is not an exclusive list, but presents the most common implementation outcomes.28 By evaluating just two outcomes of the implementation, other facets, important to the success of the implementation, might remain unobserved. But, since the majority of included studies did not evaluate the full range of Proctor outcomes, the goal of evaluating all of them might also be unrealistic. Moreover, the measurement of the full range of Proctor outcomes may not be feasible as no standardised measurement instruments exist.78 Moreover, it could also be debated which implementation outcome contributes most to successful implementation. An earlier systematic review assessed factors affecting the implementation of innovations from five levels including structural, organisational, provider, patient and innovation in order to find measures for each of the five levels.79 They found that most measures used to evaluate implementation of innovations concerns organisation, provider and innovation-level measures.79 In our review, we aimed to identify implementation success based on the Proctor et al outcomes irrespective of measurement level.28 By investigating different levels of the implementation, outcomes may have been different.80

Limitations

The goal of this integrative review was to summarise the literature on the relationship between the living lab approach and successful implementation of healthcare innovations. However, this study faced some limitations that need to be discussed. First, the quality assessment was done based on the Hawker et al quality assessment tool.38 This tool was deemed most appropriate as it is comprised of evidence from various perspectives and research methods. However, Hawker et al do not propose cut-offs for the assessment of the overall quality rating of an included study. Therefore, we applied the suggested cut-offs from the literature.39 As the tool in itself does not offer an overall quality rating, it is arguable whether these cut-offs proposed by Braithwaite et al are acceptable. We, therefore, conducted the quality assessment also by calculating the mean and came to similar results when applying cut-offs. For this additional evaluation included studies were assessed for each item from 1 to 4 with 1 indicating a good and 4 a very poor score. The cut-offs that we used for this analysis were chosen as following:.00–1.49=good; 1.50–2.49=fair; 2.50–3.49=poor; 3.50–4.00=very poor. Second, the selection of all studies reporting and evaluating at least one of the Proctor outcomes might have had an impact on the study selection. Studies were included irrespective of study design and measurement instruments used for the success of the implementation. A positive implementation outcome was reported even when included studies only offered narrative results on the impact on implementation outcomes. But since no standardised instrument for the measurement of implementation outcomes exists, we tried to elicit results as closely as possible to the results of included studies. The goal of this integrative review was to summarise all available literature irrespective of study design. Integrative reviews allow for the synthesis of information to gain a broader understanding from both qualitative and quantitative studies.29 We, therefore, believe that we selected all relevant evidence from existing literature. Third, only studies published in English, Dutch and German were included which might have introduced language bias to the study, not presenting all the evidence. However, limiting searches to English-only is still common.81 English is generally perceived to be the universal language of science and research shows no evidence of systematic bias from the use of language restrictions in systematic review-based meta-analyses.82 Not using a language restriction would have led to resource challenges with respect to costs, time, and expertise in non-English languages; however, inclusion would have contributed to ensure generalisability and reduce the risk of bias.81

Future research should focus on evaluating the full range of suggested implementation outcomes by Proctor et al for evaluating whether the living lab approach has impact on all aspects of the implementation of healthcare innovations. Furthermore, studies evaluating the living lab approach compared with a context that did not apply the living lab approach should be conducted to understand the effect or added value of applying the living lab approach. Our study showed that the living lab approach was mainly used in the development phase of healthcare innovations. Future studies should examine whether using the living lab approach in the evaluation of a new healthcare innovation or implementation can also be gainful.

Conclusions

The living lab approach seems to foster collaboration and participation of important actors in the design, development and evaluation of new innovations in healthcare. Six facilitators for successful implementation were found which can help future studies in designing living labs. The evaluation of implementation success needs to be further evaluated as currently no standardised measurement tools exist.