Systems approach to health service design, delivery and improvement: a systematic review and meta-analysis.

OBJECTIVES: To systematically review the evidence base for a systems approach to healthcare design, delivery or improvement.
DESIGN: Systematic review with meta-analyses.
METHODS: Included were studies in any patients, in any healthcare setting where a systems approach was compared with usual care which reported quantitative results for any outcomes for both groups. We searched Medline, Embase, HMIC, Health Business Elite, Web of Science, Scopus, PsycINFO and CINAHL from inception to 28 May 2019 for relevant studies. These were screened, and data extracted independently and in duplicate. Study outcomes were stratified by study design and whether they reported patient and/or service outcomes. Meta-analysis was conducted with Revman software V.5.3 using ORs-heterogeneity was assessed using I2 statistics.
RESULTS: Of 11 405 records 35 studies were included, of which 28 (80%) were before-and-after design only, five were both before-and-after and concurrent design, and two were randomised controlled trials (RCTs). There was heterogeneity of interventions and wide variation in reported outcome types. Almost all results showed health improvement where systems approaches were used. Study quality varied widely. Exploratory meta-analysis of these suggested favourable effects on both patient outcomes (n=14, OR=0.52 (95% CI 0.38 to 0.71) I2=91%), and service outcomes (n=18, OR=0.40 (95% CI 0.31 to 0.52) I2=97%).
CONCLUSIONS: This study suggests that a systems approaches to healthcare design and delivery results in a statistically significant improvement to both patient and service outcomes. However, better quality studies, particularly RCTs are needed.PROSPERO registration numberCRD42017065920. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first systematic review to provide a comprehensive and transparent synthesis of the published evidence base for a systems approach to healthcare design, delivery and improvement.

A major limitation of our study rests on the heterogeneity of the literature it seeks to synthesise, with wide variation in the settings, participants, comparators, follow-up durations and study designs.

We have conducted two exploratory meta-analyses in order to give an overview of the general direction of results, and we acknowledge that these may give artificial numerical precision which may not be warranted.

This benefit must be interpreted and applied with care because the evidence mostly comes from before-and-after study designs, with inherent confounding factors of unknown magnitude and direction.

Several included studies reported both the potential of a Hawthorne effect and the existence of other interventions at the time of their study which may have contributed to their observed outcomes.

Introduction

The 20th and 21st centuries have witnessed the development of highly effective healthcare technologies, diagnoses and interventions.1 2 Nonetheless, there remains a pressing need for improvement in both the quality and safety of care delivery.3–5 This is often attributed to several factors including multimorbidity,6 the complexity of healthcare delivery7 and a variety of cultural and organisational challenges.8 9 Drawing on the experience of fields such as engineering and design a ‘systems approach’ to improvement has been advocated, that recognises the interacting components of healthcare delivery, the people involved, as well as planned, considered and adaptive iterative implementation.10–16 However, there has not been a systematic review of the evidence base for such an approach within the healthcare literature to date.

Modern healthcare systems are striving for integrated, patient-centred, effective and efficient care,17 but the lesson from engineering is that such systems do not happen by accident; they need to be planned, designed and built.18 Understanding what this process might look like has been explored with reference to the literature on patient safety,19 Human Factors and Ergonomics (HFE),20 general practice,21 the well-being of healthcare workers22 and public health.23 These reviews, while useful, are limited in their scope and employ narrow views of a systems approach.

The primary objective of this study is to review, comprehensively, the usefulness of a systems approach to healthcare improvement. There were no limits on language, participant types, outcome types or any particular healthcare domain.

Definition of a systems approach

Defining a systems approach is challenging. The approach has origins in a variety of disciplines, which have both diverged and converged over the past century. These range from mathematics to social science, and span both the physical and biological sciences.24 In order to arrive at a definition that we could operationalise for the purpose of this systematic review, the team reviewed definitions of a systems approach including Clarkson et al,10 Maier and Rechtin,25 Chen26 and the NASA systems engineering handbook.27 As a result, we developed a shared understanding of a system, at its fundamental level, as:

A collection of different elements (or things) which together produce results unachievable by the individual elements on their own.28

Our working definition of a systems approach, which has been informed by Clarkson et al,10 is as follows:

A systems approach to healthcare improvement is a way of addressing health delivery challenges that recognises the multiplicity of elements interacting to impact an outcome of interest and implements processes or tools in a holistic way.

This view of a systems approach integrates perspectives on people, systems, design and risk in a way that is applicable to healthcare systems across all scales from local service systems through to organisational, cross-organisational and national policy levels.

Methods

This systematic review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standard.29

Included were published primary research studies involving any patients in any healthcare setting where a systems approach was compared with concurrent or historical comparators—usual care or a non-systems approach, which reported numerical results for both groups for any outcomes relevant to the study being conducted. Excluded were conference reports or other unpublished studies, studies without clear evidence of a systems approach being used, studies without any type of comparator or studies without quantitative outcome data for either group.

Information sources

We searched the following databases with no limits on date of publication: Medline, Embase and HMIC (via OVID), Health Business Elite, PsycINFO and CINAHL (via EBSCO), Web of Science and Scopus. The search was first conducted in August 2017 and repeated on 28 May 2019. It was repeated for a second time on 24 July 2020, but we found very little additional evidence, in particular nothing that would affect the meta-analysis direction and the conclusions of the systematic review. As a result, the analysis and results presented here are based studies up to May 2019. There were no limits on language, participant types, outcome types or any healthcare domain. The full search strategy including specific search strings are provided in online supplemental file 1.

Study selection process

We used a structured, two-stage approach to determine inclusion. The first stage involved a title/abstract review of citations after removing duplicates. The second consisted of a full-text review of the 107 papers identified as potential for inclusion.

For the title/abstract review stage, three pairs of researchers looked at a third of the records each. Studies were selected for inclusion or rejection independently by each researcher, and with differences resolved first within the pair, and then within the whole team where the pair could not agree.

The full-text review stage applied the definition of a system and of a systems approach as stated above. Researcher pairs individually reviewed studies for inclusion or exclusion based on the following two questions:

Does the study identify a clear problem framed in a systems context and demonstrate the use of a systems approach, in some way? AND

Does the study have an appropriate design to address the research question?

Question one excluded any study which did not in some way demonstrate a systems approach in its formulation and/or implementation of an improvement intervention, while question two excluded all protocols, conference abstracts, systematic reviews, reviews, editorials and any paper with no primary research or no comparator arm.

Following the individual assessment, members of each pair discussed their results to arrive at a consensus on which studies to include. As a final check for all included studies, the team assessed each study against aspects of our working definition of a systems approach. Studies were assessed on a binary scale (0 or 1) as to whether they demonstrate a consideration of systems in the development of an intervention or in the implementation of the intervention, use of design and a consideration of risk. The outcome of this assessment is shown in table 1 in online supplemental file 2. A full list of excluded studies with reasons for exclusion is also provided in online supplemental file 3. Our method is also summarised using the PRISMA systematic review process shown in figure 1.

Table 1

Characteristics of included studies

Study	Country	Health setting	Population	Intervention	Study type	Follow-up duration	Participants		Outcomes
							Before	After
Afsar-manesh et al, 201731	USA	Whole hospital—in general medicine, general surgery, neurosurgery, paediatrics, orthopaedics	Staff and patients	Comprehensive Lean methodology	Before and after	18 months	NR	NR	Patient
Allaudeen et al, 201737	USA	University-affiliated department of veterans’ affairs medical centre	Emergency Department patients and staff	Lean-based multidisciplinary initiative and PDSA	Before and after/concurrent control	3 years	NR	NR	Service
Anderson et al, 201743	USA	The University of Colorado hospital academic medical centre	Geriatrics patients and staff	‘Stepwise framework for implementing a comprehensive geriatrics hip fracture programme’ involving twelve steps	Before and after	17 months	154	117	Patient /service
Bell et al, 201844	UK	Eight acute NHS hospital trusts and 12 local authority areas in North East England	Pregnant women	‘BabyClear’ – a complex intervention comprising a package of measures designed to support implementation of national guidance	Before and after	4 months	NR	NR	Patient/ service
Bhatt et al, 201441	USA	Academic Medical Centre	Nursing staff	ACGME Core Competency of Systems-based Practice	Before and after	17 days	13	17	Service
Bhutani et al, 200642	USA	Semiprivate urban birthing hospital	Babies discharged as healthy patients’ parents, paediatricians, paediatric nurses, home care nurse agencies	Systems approach to Clinical Condition Management	Before and after	12 months	32278186	818611 995	PatientService
Bowen et al, 201645	USA	Grady Memorial Hospital—single centre, hospital, stroke centre	Stroke patients	Multistakeholder process mapping to inform problem identification involving value stream mapping	Before and after	32 days	75	88	Service
Bradley et al, 201148	Ethiopia	Primary care in Rural Ethiopia	Primary care patients	The Ethiopian Millennium Rural Initiative	Before and after	18 months	140	140	Service
Catchpole et al, 201432	USA	Nonprofit, academic tertiary care medical centre	Trauma patients	A multidisciplinary systems analysis informed by Human Factors Engineering, Systems Engineering Initiative for Patient Safety (SEIPS) and PDSA (iterative)	Before and after	5 months	1472	13107	Service
Chandrasekar et al, 201740	UK	A university hospital in the UK	All medical inpatients at a single UK hospital	‘Quality Improvement Methodology’ including driver diagrams, pareto charts and statistical process analysisAlso includes a range of interventions involving risk assessment tools, early identification using automated alerts, development of an intervention bundle, formation of an outreach support team staff engagement and patient and family empowerment.	Before and after	34 months	NR	NR	Patient/ service
Cochran et al, 201838	USA	Franciscan Health Indianapolis—a general medical and surgical centre	Patients, nurses and a team of emergency room and system engineering specialists	Collective System Design (CSD) methodology involving PDCA – a systems engineering methodology, which recognises systems as the amalgamation of four key elements: Work/Actions, Structure, Thinking and Tone.	Before and after	8 months	NR	NR	Patient/ service
DeFlitch et al, 201539	USA	A suburban, tertiary care, academic emergency department, with paediatric and adult level one trauma	Patients and staff	‘Engineering techniques’ including defining a study team, process mapping, Discrete Event Simulation modelling and detailed design considerations leading to the Physician Directed Queuing (PDQ) model.	Before and after	3 years	NR	NR	Patient/ service
Dennerlein et al, 201749	USA	Hospital-wide—at two hospitals	Direct patient care workers	A broad stakeholder engagement, new lifting equipment across hospital, new processes and group training and one-to-one coaching and mentoring for staff	Before and after	12 months	21492348	21312414	Service
Gupta et al, 201872	USA	Parkland health and hospital system—a large public hospital in the USA	Healthcare staff and patients	A multidisciplinary team delivering PDSA including process mapping	Before and after	6 months	36	28	Service
Hathout et al, 201350	Canada	Province-wide, Manitoba, Canada	Healthcare staff	Stakeholder engagement, problem exploration, process mapping, exploration of systems drivers and value and objectives of services	Before and after	18 months	NR	NR	Patient/ service
Heymann et al, 200451	Israel	Maccabi Healthcare services, a Health Maintenance Organisation serving 1.5M patients	Healthcare staff	Previously developed Systematic Inventive Thinking. Bases on ‘Creativity as an exact Science’	Before and after	14 months	1000	1000	Service
Hultman et al, 201673	USA	Academic medical centre of the University of North Carolina Hospitals	Healthcare staff and patients	Lean-Six Sigma—using standard DMAIC model	Before and after	24 months	39 (27)	46	Patient/ service
Huntington et al, 201274	Philippines	Health systems reform in a province in the Philippines involving two tertiary hospitals,20 first level referral health facilities.Twelve rural health units.One village health station	Women’s health teams	National initiative - National Safe Motherhood Programme. Seems influenced by WHO health systems strengthening principles	Before and after/concurrent control	4 years	16 535NR	15 789NR	Patientservice
Hwang et al, 201752	South Korea	Multiple institutions in the chain of survival of cardiac arrest patients—community to hospital	Cardiac patients	System-wide Cardiopulmonary Resuscitation (CPR) programme for out-of-hospital Cardiac Arrest (OHCA) patients developed by lead Hospital.	Before and after	12 months	182282	282182	PatientService
Kane et al, 201960	USA	A tertiary care hospital in Baltimore, USA	Nursing bed managers, transfer line operators, patient pathway coordinators	‘A systems engineering approach’ involving a steering group consisting of hospital Chief Executive Officer (CEO), Chief Information Officer (CIO), Chief Operating Officer (COO), Vice President for medical affairs, Director of nursing, and project leaders. Involved working with external supplies and use of DES and ABS.	Before and after	Can’t tell	NR	NR	Service
Khan et al, 201846	Canada/USA	Paediatric inpatient units in seven North American hospitals in USA and Canada	Patients, parents or caregivers, nurses, medical students and residents	Co-production of intervention to standardise the structure of healthcare provider-family communication on family centred rounds.	Before and after	3 months	9471574	8901532	Patient
Kottke et al, 201656	USA	Private, five-clinic primary care practice	Coronary Heart Disease (CHD) patients	Complex Adaptive Systems principles	Before and after	6 months	529529	511511	PatientService
Lick et al, 201153	USA	Community-based centres of excellence	Cardiac arrest patients	‘Take heart America programme’: Community-based initiative	Before and after	6 months	247247	106106	PatientService
Loh et al, 201775	Singapore	National tertiary specialist hospital	Cataract surgery patients	Systems Engineering Initiative for Patient Safety (SEIPS) Model and PDSA	Before and after	6 months	6111	39 390	Service
McGrath et al, 201959	USA	US Academic Health Centre	Patients and staff including nurses, nurse assistants, occupational therapists, physical therapists, physicians	System-level design and analysis involving system design and validation, installation and education, operation and performance measurement	Before and after/concurrent control	5 months	557	678	Service
McKetta et al, 201657	USA	The Cardiac Centre at a Children’s Hospital	Cardiac centre staff	A Discrete Event Simulation together with traditional QI involving a multidisciplinary team using a four-step framework—Define, Diagnose, Test and Implement, and Sustain. Including PDSA	Before and after	4 months	135	138	Service
Moran et al, 201847	UK	UK NHS in England and North Wales	Population of England and wales	Trauma systems—Systematic trauma care on a national basis.	Before and after	4 years	44 05941 149	17 95617 092	PatientService
New et al, 201634	UK	The Orthopaedic trauma theatre of a UK hospital Trust	Theatre staff	A two-step intervention—1 hour Lean training followed by 6 months coaching. Training covered Lean principles Muda, Poka-Yoke, Genchi Genbutsu, Kaizen, flow, Just-In-Time (JIT), respect and teamwork, process mapping, PDCA cycles	Before and after/concurrent control	6 months	45025	56717	PatientService
Rateb et al, 201135	Egypt	Egypt HIO/community,Medical fitness testing.	HIO doctors, nurses, admin staff, customers	Business Process Re-engineering focusing on Structure, Process and Outcome. Systems approach appears to mean everything from building renovation to customer and staff satisfaction.	Before and after	Can’t tell	251101	251101	PatientService
Rothemich et al, 201054	USA	16 primary care practices	Adult smokers, family physicians,	Called QuitLink Intervention. Limited details provided.Described as using paper-based, systems approach to identify smokers, provide advice to quit, and assess willingness to quit. Includes supporting willing smokers too access quitlines and communicate feedback from quitline to clinicians	Concurrent control (RCT)	1 month	958	857	Service
Rustagi et al, 201636	Ivory Coast, Kenya and Mozambique	Mother-to-child HIV transmission prevention services in three countries in Africa—Cote d’Ivoire, Kenya and Mozambique	Healthcare staff and patients	The Systems Analysis and Improvement Approach—a 5-step, iterative package of systems analysis and improvement tools developed using multiple systems engineering techniques including continuous quality improvement.	Before and after/concurrent control (RCT)	9 months	17	18	Patient
Ryan et al, 200655	UK	Manchester Alcohol Service (MAS)In-patient detoxification service	Detox service users	A whole systems approach to alcohol services—A collaborative working between multiple organisations	Before and after	Can’t tell	171	2754	Patient
Shultz et al, 201576	USA	5 Family Medicine Clinics and 4 Internal Medicine Clinics (as control)	Physicians and staff	Sequential and linked PDSA/Adjust cycles.A consensus-based framework that addresses the process of care	Before and after/concurrent control	2 years	67 914	67 914	Service
Srinivasan et al, 201777	USA	Emergency department and inpatient unit of a 280-bed tertiary care, free-standing children’s hospital	1–23 months babies and parents	Driver Diagram plus three cycles of PDSA involving stakeholder surveys focusing on changing clinician behaviour through both education, re-enforcement and encouragement.	Before and after	3 weeks	22111486	9111597	Patient/ service
Tetuan et al, 201758	USA	Integrated healthcare systems comprising primary and specialty clinics, and a 568-bed acute care hospital.	Nurses	Systems Thinking Education Programme	Before and after	12 months	1652	1998	Service

ABS, Agent-Based Simulation; ACGME, Accreditation Council for Graduate Medical Education; DES, Discrete Event Simulation; DMAIC, Define, Measure, Analyze, Improve, Control; HIO, Health Insurance Organisation; NHS, National Health Service; NR, Not Reported; PDCA, Plan-Do-Check-Act; PDSA, Plan-Do-Study-Act; QI, Quality Improvement; RCT, Randomised Controlled Trial.

Figure 1

PRISMA systematic review process. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Data collection

A template for data extraction was developed by the research team working through samples of the selected papers to identify relevant fields and tables appropriate to the study question. The data extraction process was designed to include an element of quality control and minimisation of researcher bias. The lead author initially extracted data from all included studies using the agreed template, with other team members each assigned a subset of these to independently corroborate.

Data were extracted into tables as listed below, and all included in online supplemental file 2. Patient outcomes and service outcomes are each separated into two tables according to study design (see online supplemental table S2–S5b). Online supplemental tables S6–S8 are the results of applying the Critical Appraisal Skill Programme (CASP)30 questions to included studies:

Study source, Country, year and aspects of systems approach.

Characteristics of studies (population and intervention).

Characteristics of studies (design, baseline type, blinding and funding source).

Patient outcomes

Patient outcomes for studies with before and after design.

Patient outcomes for studies with concurrent design.

Service outcomes

Service outcomes for studies with before and after design.

Service outcomes for studies with concurrent design.

CASP questions for appraisal of cohort Studies.

CASP questions for appraisal of case–control Studies.

CASP questions for appraisal of randomised controlled trials (RCTs).

Examples of patient outcomes include numbers of vaccinations received, numbers of medication events and time to death. Examples of service outcomes include appointment delays, customer flows and time to treatment received. We did not include every outcome as this was impossible. We also did not use summary outcomes as this will give undue weighting to some studies compared with others. Outcomes were selected based on their relevance to the overall objective of the respective studies.

Patient and public involvement

Due to the focus of this review on synthesising evidence within the academic literature, patient and public involvement was not applicable.

Data analysis

Review manager (V.5.3, The Cochrane Library) was used for the meta-analyses using a random-effects model due to the heterogeneity of participants, interventions and outcome measures. Meta-analysis was conducted for service outcomes and patient outcomes separately where the categories below had the highest number of studies. Categories were before-and-after studies, studies with concurrent controls and continuous versus categorical versus time-to-event data. The highest numbers of studies for both service and patient outcomes were the before-and-after studies so this category was used in both meta-analyses. Heterogeneity was assessed using the I2 statistic, using standard thresholds. Risk of publication bias was assessed by use of a funnel plot primarily using the service outcome studies and adding the patient outcome results for studies not already in the service outcome meta-analysis.

Risk of bias for all studies was assessed by two researchers independently using CASP checklists.30 These were chosen because they have a suite of checklists appropriate for different study designs. Differences were resolved through a consensus process. The CASP checklist for cohort studies, case control and RCTs were applied accordingly. The checklists consist of 11 or 12 questions in three sections—study validity, study results and local value of results. A complete PRISMA checklist (online supplemental file 4) is also included to illustrate adherence to the review process.

Results

Our initial search found 11 463 records published prior to August 2017 and an extended search in May 2019 found a further 3081 records. After deduplication there were 11 405 citations including two records added from personal sources. Of these, 11 298 records were excluded after the scanning process, leaving 107 full texts. Included were 35 studies, out of which 23 provided sufficient data for the two meta-analyses conducted (figure 1).

Of the 35 included studies, 28 (80%) had a before-and-after design only. Six studies had both a before-and-after and concurrent design (including two RCTs). Summary characteristics of included studies are presented in table 1. Studies excluded at the full-text review stage, with reasons for exclusion, are provided in the online supplemental file 3.

There was considerable diversity in how a systems approach was conceptualised and implemented in the included studies. This diversity in approaches may be categorised in three ways:

A comprehensive implementation of traditional tools and approaches such as Plan-Do-Study-Act (PDSA), Lean, Human Factors Engineering, WHO health systems strengthening principles, SEIPS model, Business Process Re-engineering, Structure- Process- Outcome and various combinations of these.31–40

A focus on the breadth of coverage of the intervention, involving a wide range of stakeholders from patients, communities, multiple departments including consideration of physical structures.41–55

The application of standard systems concepts such as systems thinking and complex adaptive systems theory.56–60

Almost all included studies showed a benefit for using a systems approach for almost all the outcomes. The exceptions were New et al34 (service outcome, concurrent control) and Dennerlein et al49 (service outcome, concurrent control). Most of the factors reported as contributing to success were related to people. These were expressed in the form of engaging with stakeholders, taking a team-based approach, enhancing communication, adopting a collaborative approach, patient- centredness and physician-centredness. Similarly, difficulty in measuring impact and the inability to generalise to other contexts emerged as the most significant limitations.

We included two RCTs in our systematic review. Both reported significant improvements in outcomes favouring a systems approach. Rustagi et al36 randomised 36 health facilities in Cote d’Ivoire, Kenya and Mozambique to usual care or ‘a systems engineering intervention’ stratified by country and volume. They found that antiretroviral (ARV) coverage for HIV positive women increased threefold in intervention facilities compared with control facilities while HIV-exposed infants screening increased 17-fold. Similarly, Rothemich et al54 randomised 16 practices into intervention (8) and control (8) groups to determine whether a systems approach enhances smoking cessation support in primary care practices. The study concluded that a systems approach to identifying smokers, advising, assessing readiness to quit and referral to supporting agencies, led to statistically significant increases in cessation for patients irrespective of gender, compared with traditional tobacco-use vital sign screening alone.

Two exploratory meta-analyses were conducted on categorical outcomes reported in before-and-after studies; one on patient outcomes (figure 2), and one on service and resource use outcomes (figure 3).

Figure 2

The impact of a systems approach on patient outcomes—before-and-after studies. M-H, Mantel-Haenszel

Figure 3

The impact of a systems approach on service and resource use—before-and-after studies. M-H, Mantel-Haenszel

Exploratory meta-analysis suggests that a systems approach significantly improves both patient outcomes (n=14, OR=0.52 (95% CI 0.38 to 0.71) I2=91%) and service outcomes (n=18, OR=0.40 (95% CI 0.31 to 0.52) I2=97%).

Heterogeneity was very high. The funnel plot (figure 4) is unclear regarding publication bias. If anything, it might suggest that small studies with very positive results are missing, rather than those with null results.

Figure 4

Funnel plot using combined service and patient outcome results.

It is important that the above results are interpreted with the heterogeneity and the quality of the included studies in mind. The two included RCTs both had reasonably high quality. The five cohort studies with concurrent controls varied between good and fair quality. The before-and-after studies which made up 80% of included studies varied widely in quality, ranging from good to very poor. Details of the quality assessment results are included in the last three online supplemental table S5–S7.

Discussion

Our novel systematic review with exploratory meta-analyses suggests that the use of a systems approach to improving care results in significant benefits for both patient and service outcomes. There were two RCTs included that individually found statistically significant improvements in outcomes associated with the use of a systems approach. These findings, together with the observation that the majority of studies had a before-and-after design, present a challenge in interpretation of results in relation to what is usually considered good quality evidence. In addition, we observed a number of factors, which may support success in the use of a systems approach as reported by the included studies. To our knowledge, this is the first systematic review that has endeavoured to conduct a comprehensive synthesis of the evidence base for a systems approach to healthcare improvement.

This review adds to a growing number of systematic reviews apparently motivated by the desire to find evidence for what works in healthcare improvement. Narrative reviews19–23 of a systems approach in healthcare have focused on specific health issues such as patient safety, Human Factors and Ergonomics (HFE) in healthcare, primary care, well-being of health workers and public health. Though these generally demonstrate value of a systems approach, they lack a rigorous and comprehensive assessment of the evidence base for this. Other systematic reviews have been conducted on most of the major healthcare improvement methodologies including Lean,61 Six Sigma,61 62 PDSA,63 Statistical Process Control (SPC)64 and Quality Improvement Collaboratives (QIC),65 with mixed results. DelliFraine et al61 in their review of both the Lean and Six Sigma methodologies concluded that there is very weak evidence that either of the methods improves care. However, the review did not provide a meta-analysis of the studies identified and only focused on studies between 1999 and 2009, thus limiting its value. Taylor et al66 in their review of PDSA found poor compliance with the original principles of the methodology but did not aim to assess the impact of the method on outcomes. In the review of SPC, the authors found considerable benefits of using the approach to monitor and control health processes, though they acknowledge some limitations exist. Wells et al65 in their review of QICs reported significant improvements in process and patient outcomes. Their review reported outcome measures from included studies but stopped short of a full meta-analysis. Our findings are also consistent with the expectations of positive impact from the several publications that have called for a systems approach to tackling the challenges of modern health delivery systems.10–16 There is, clearly, considerable interest in assessing the evidence base of various improvement methodologies, however, existing systematic reviews have not been comprehensive enough and lack focus on patient and service outcomes.

Though the current review focuses on a systems approach to improvement, we believe this represents the most comprehensive systematic review and meta-analysis so far for evidencing the effectiveness of an improvement methodology. This is because we had no limits on date of publication, health setting, study type or participant types. We wanted the results to be relevant to a wide range of healthcare improvements contexts. However, one may object to our decision to combine very heterogeneous studies as we have done because of the differences in clinical settings and outcomes being measured. We reasoned that the results of a combined study would be more useful to the healthcare community, practitioners and policy-makers than an issue-specific systematic review. Moreover, several of those already exist, although not as rigorous. The inclusion of two RCTs in this review further strengthens the results. Though limited in number, both RCTs report statistically significant improvements in outcomes following the implementation of a systems approach.

Limitations

The major limitation of our study rests on the heterogeneity of the literature it seeks to synthesise, with wide variation in the settings, participants, comparators, follow-up durations and study designs. We have sought to mitigate this using a clearly articulated definition of a systems approach, and a structured, rigorous, approach to synthesising the available evidence. We have conducted just two meta-analyses in order to give an overview of the general direction of results. We acknowledge that the estimated effect size gives an artificial precision which may not be warranted. The heterogeneity of meta-analysis results is to be expected, given the wide variation in participants, settings, interventions, comparators and outcomes. This exploratory meta-analysis can only indicate that a systems approach appears to be beneficial. This benefit must be interpreted and applied with care because the evidence mostly comes from before and after study designs, with inherent confounding factors of unknown magnitude and direction. There is also a significant risk of publication bias, and several included studies also reported both the potential of a Hawthorne effect and the existence of other interventions at the time of their study which may have contributed to their observed outcomes. The fact that we selected outcomes based on their relevance to the overall objectives of the studies included may introduce another level of bias if authors framed their objectives based on what they wanted to publish.

Implications for further research

The engineering sector is one that has excelled in the application of a systems approach.18 The experience of the Systems Engineering community is that the value of a systems approach—in terms of quality of the resulting system, reduction in cost, delivery on time, customer satisfaction—corresponds to the extent to which a project or organisation commits to the approach.67 68 This has implication for our findings in this review. It helps raise a number of questions that present opportunities for future research. For example, what are the different ways in which a systems approach is implemented in healthcare? Is there an association between the time and resource invested in a systems approach and the impact on patient and service outcomes? If so, what is the optimum level of investment?

Another opportunity for future research is a comparative review which assesses the impact of all improvement initiatives against those explicitly adopting a systems approach if more certainty of the value of the approach is desired. Given the volume of literature involved in such a comparative review, this would represent a significant undertaking. Studies are also needed that adopt better study designs such as RCTs or, if necessary, develop alternative ways of understanding and achieving sufficiently robust evidence for a systems approach to healthcare design and delivery. This is a point pertinent to all improvement efforts, where the traditional medical model of the randomised controlled trial is rarely appropriate, but the need to generate convincing evidence remains pressing.

Policy implications

We have argued from the start that there has been a growing recognition of the potential value of a systems approach to healthcare improvement over the past two decades. Most of this recognition has been at the policy level, involving the WHO,69 the Institute of Medicine in the USA,3 4 12 the Department of Health in the UK70 71 and more recently, through a joint initiative between the Royal Academy of Engineering, Royal College of Physicians and the Academy of Medical Sciences.10 However, to support further research and increased practice of a systems approach in health and care, policy-makers need to understand the evidence base. Though several success stories and domain-specific reviews exist, a comprehensive review of the evidence across the healthcare literature has been lacking. Our review may, therefore, become invaluable to policy-makers who have found the argument for a systems approach conceptually appealing but also desire to see the evidence of what difference such an approach can make to patient and service outcomes. In addition, the references taken individually may serve as examples of real-world applications of a systems approach to healthcare improvement.

Conclusions

In summary, we have argued that a systems approach to healthcare has been championed increasingly in the health and care literature and in a variety of grey literature reports and position documents. We provide the first attempt to comprehensively explore the evidence base through a systematic review and meta-analysis. The results provide reasonably clear evidence that a systems approach to addressing health delivery challenges may lead to significant improvements in both patient and service outcomes.