Evaluation protocol of the implementation of a learning healthcare system in clinical practice: the Connected Health Cities programme in the north of England.

INTRODUCTION: The 'learning healthcare system' (LHS) has been proposed to deliver better outcomes for patients and communities by analysing routinely captured health information and feeding back results to clinical staff. This approach is being piloted in the Connected Health Cities (CHC) programme in four regions in the north of England. This article describes the protocol of the evaluation of this programme. METHODS AND ANALYSIS: In designing this evaluation, we had to take a pragmatic approach to ensure the feasibility of completing the work within 1 year. Furthermore, we have designed the evaluation in such a way as to be able to capture differences in how each of the CHC regions uses a variety of methods to create their own LHS. A mixed methods approach has been adopted for this evaluation due the scale and complexities of the pilot study. A documentary review will identify how CHC pilot study deliverables were operationalised. To gain a broad understanding of CHC staff experiences, an online survey will be offered to all staff to complete. Semi-structured interviews with key programme staff will be used to gain a deeper understanding of key achievements, as well as how challenges have been overcome or managed. Our data analysis will triangulate the documentary review, survey and interview data. A thematic analysis using our logic model as a framework will also be used to assess progress against the CHC programme deliverables and to identify recommendations to support future programme decision-making. ETHICS AND DISSEMINATION: Ethical approval was granted by The University of Manchester Ethics Committee on 24 May 2018. The results will be actively disseminated through peer-reviewed journals, conference presentations, social media, the internet and various stakeholder/patient and public engagement activities. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study will represent the largest evaluation ever conducted to examine the barriers, facilitators and lessons learnt for creating and piloting learning healthcare systems in four regions in the north of England.

The use of a mixed methods approach will mitigate the risk of not having baseline outcome measures at the start of the pilot study period, as well as the risk of sampling, recruitment and participation bias.

The use of purposive sample, while based on access to the subject group most appropriate for taking part in this study, will elevate the risk of self-selection bias.

Background

The UK is experiencing rapid changes to its population: an ageing population, increased life expectancy and changing patterns of chronic diseases have led to an increased demand in health and social care services.1 At the same time, the amount of health data being digitally collected and stored is vast and expanding rapidly,2 while the technology and analytical tools needed to analyse large datasets have also been developed.3 Therefore, there is the potential for using routinely collected health data to drive forward improvements in health outcomes.4–6

The Connected Health Cities (CHC) (https://www.connectedhealthcities.org/) programme is a U.K government-funded programme that aims to create learning healthcare systems (LHSs) across the north of England. An LHS is defined by the Institute of Medicine7 as a system in which

’science, informatics, incentives, and culture are aligned for continuous improvement and innovation […] with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience.’

Friedman et al 8 describes a common cycle process that can be found in all LHSs. This is characterised by five steps divided between an afferent and an efferent side8:

Afferent side

Assemble the data from various sources.

Use a range of analysis on the data.

Interpret the findings.

Efferent side

Feed the findings back into the system in many formats.

Change practice.

Building on this notion of an LHS, a CHC is a civic partnership in which health and care services, science, technology and work culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as a by-product of delivering care. At the centre of a CHC is a secure information system called an ‘Ark’, which will provide a trustworthy, regional combinatorial innovation centre for health and social care data analysis, providing timely and actionable information for the care of the population it serves (see figure 1).

Figure 1

Connected Health City: Ark-enhanced information flows.

This protocol describes the design of the evaluation of the CHC programme which aims to answer the following objectives:

To evaluate progression and early impact of each CHC region against seven Department of Health and Social Care (DHSC) deliverables.

To identify the benefits, additionality and added value of the CHC programme.

To identify the challenges of implementing an LHS in four regions in the north of England and how these have been overcome.

To assess the level of input required from staff, resources and approvals (such as information governance) to create each regional Ark and care pathway.

The outline of this paper is as follows: first, we provide a summary of the overall CHC programme, outlining the deliverables for the programme and pilot study, as well as the eight pathways that have been chosen for inclusion in the evaluation. Then, we describe the analytical framework that underpins the evaluation, discussing the formulation of a logic model and programme deliverables. Next, we discuss the logic model and the issues that need to be considered in formulating the research design. Next, we present a summary of the three data collection methodologies to be used in our evaluation: the documentary review, questionnaire for all CHC staff and a semi-structured interview. This is followed by a description of how we will analyse the data collected. Finally, our concluding remarks summarise the key issues and approaches used in our evaluation.

Design of Connected Health Cities programme

The CHC programme is a Northern Health Science Alliance-led programme delivered by a consortium of academics and NHS organisations, including NHS Trusts and Clinical Commissioning Groups, across the north of England. It is being funded by the DHSC to assist in the delivery of the UK government’s commitment to reducing healthcare need, reducing inequalities and constructing the ‘Northern Powerhouse’. The CHC programme covers four regions: Greater Manchester, Yorkshire, North West Coast and North East and North Cumbria. Each region has been tasked with establishing an LHS, using patient data to create and test innovative solutions for a variety of clinical pathways. This also includes the development of a central development hub to oversee the overall programme of work in relation to seven deliverables (table 1): establishment of data sharing strategy and agreements for each region; establishment and delivery of governance arrangements for the sharing and usage of data for each region; workforce arrangements optimised and Continuing Professioal Developemnt (CPD) requirements identified; creation of Arks as analytical platforms; pathway analysis, variation assessment and improvements identifications; frameworks and integration with R&D partners and the production of a business model suitable for scaling and sustainable for delivery in the NHS.

Table 1

The CHC Programme deliverables

Deliverable	Description of deliverable
Deliverable 1	The establishment of data sharing strategies and data sharing agreements for each CHC region.
Deliverable 2	The establishment and delivery of governance arrangement for the sharing and usage of data for each CHC region, across the North and the UK.
Deliverable 3	The optimisation of workforce arrangements, including the identification of long-term Continuing Professional Development (CPD) requirements and the establishment of new skill bases.
Deliverable 4	The creation of the Ark as an analytical platform for investigating linked data.
Deliverable 5	The analysis of eight care pathways, identification of any pathway variations and proposals for any improvements, if possible.
Deliverable 6	The creation and implementation of frameworks for potential integration with R&D partners and the future rising of Foreign Direct Investment.
Deliverable 7	The production of a CHC business model suitable for scaling across the North and sustainable for delivery in the NHS.

The CHC programme has over 16 different care pathways in the process of delivery. However, while the number of pathways varies for each region, the CHC programme was tasked with developing at least two pathways per region. Our funders requested that eight care pathways were included as part of the evaluation process. Table 2 shows the four CHC regions in relation to the eight care pathways with a brief overview of the work undertaken for each pathway. The eight pathways included in this evaluation were selected to showcase the types of data that could be analysed to inform the pathways of a range of health issues.

Table 2

Description of care pathways included for evaluation, by region

CHC region	Title of care pathway	Objectives of care pathway	Description of care pathway
Connected Yorkshire	Supporting community care and reducing demand on A&E services	To link de-identified routine NHS data to describe a detailed profile of patient demand across both prehospital, primary care and hospital emergency and urgent care settings in Yorkshire.	To collect routine NHS data from a number of EUC providers and link the data to provide a coherent picture of EUC demand.
	Safer prescribing for frailty	To reduce inappropriate polypharmacy for people with frailty.	To work with GPs to change behaviours related to deprescribing for older people with moderate or severe frailty as identified by electronic Frailty Index scores. This includes developing interventions using which apply evidenced tools to support deprescribing.
Greater Manchester	BRIT—Using data to tackle antibiotic resistance	To provide the NHS and clinical care teams with better information on what is happening and who is getting antibiotics. To assist in determining whether the use of antibiotics is reasonable given local resistance patterns to antibiotics.	Analysis of patient records from GPs for effectiveness of antibiotic prescribing in general practices. This includes the development of a DataLab feeding back advanced analytics to clinical staff and policy makers and the evaluation of interventions to optimise prescribing.
	Using technology and data to improve the diagnosis and treatment of stroke	Improve the recognition of stroke by paramedics to maximise the proportion of acute stroke patients taken directly to a specialist stroke centre for timely expert care and minimising the number of non-stroke patients entering the stroke pathway. Provide timely and focused referral to neurosurgery for patients in Greater Manchester with stroke caused by a brain haemorrhage. Ensure that all patients get all the right treatments that they need to reduce the risk of another stroke when they are discharged from hospital.	To improve stroke recognition by paramedics by linking ambulance data to data at Salford Royal; using primary and secondary care data to create a large cohort of stroke and TIA patients for creating a predictive model of patients who are at high risk of stroke; and using acute trust data to identify predictive factors of early deterioration and death.
North East North Cumbria	Predictive modelling for unplanned care	To develop predictive modelling tools for unplanned care forecasting to support demand management and service planning in relevant health and social care services.	To produce statistical models that can be used by health/local authority/other analytics teams to produce daily forecasts up to 6 months in advance with the pertinent associated uncertainties and variations in urgent and emergency care.
	SILVER: Smart Interventions for Local Vulnerable Families	To develop data sharing agreements to allow the linking of existing health data across multiple health agencies via one platform that provides recommendations to key workers.	To link data across multiple agencies including health (physical and mental), social care, criminal justice, housing and education to develop a more complete Learning Health System.
North West Coast	Development of a learning system for alcohol	To be able to inform health professionals about local clinical care. To define best care or treatments, implement and demonstrate benefits.	Improving the way information is collected, analysed and shared between agencies and service users to bring opportunities for news was to respond collectively.
	Development of a learning system for unplanned care	To improve how data are used to enhance patient care admitted to hospital for emergency care.	Linking NHS data with social services data to improve the care pathway for patients with COPD and epilepsy.

Logic model

The Centre for Disease Control9 framework for evaluation mentions ‘logic models’ as useful tools to help describe a programme or policy.9 10 Logic models can be visualised as a sequential ‘if-then’ process.11 This can be used as a basis for planning an evaluation strategy as it allows for the identification of the various steps that need to be fulfilled before one can expect to see the desired outcome from a programme or policy.12 13

The evaluators have developed a logic model to assist in assessing the CHC programme against the seven deliverables (figure 2). Logic modelling is a tool that can be useful in the development of monitoring and evaluation plans, identifying short, medium and long-term outcomes that are linked to key activities of a programme.14 Throughout the first month of the evaluation, meetings were held with CHC staff from each of the different regions to gain an overview of the work being completed in relation to the CHC programme deliverables. Information from these meetings was combined with a retrospective documentary review to formulate the logic model featuring input, output, outcome and impact stages. The logic model ensured that there was a consistent and systematic means in the design of the evaluation. This logic model is expected to change throughout the duration of the evaluation as data are gathered and other factors are found that have contributed to the CHC programme.

Figure 2

Logic model for the Connected Cities (CHC) pilot study evaluation. CHC, Connected Health City; PPI, patient and public involvement; TREs, Trusted Research Environments.

Inputs

Certain resources are needed to operationalise the CHC programme of work. These include the recruitment of staff, such as statisticians, clinicians, qualitative researchers and software engineers. Other resources included financial input and any infrastructure needed such as buildings, computers and software.

Outputs

If the CHC programme has access to the inputs, then they can be used to accomplish the planned outputs. Outputs were divided into two distinct areas: activities and participation. Activities include the creation of Trusted Research Environments (TRE), putting in place regional and pathway governance arrangements, creating analytical platforms, identification of care pathways, patient and public involvement (PPI) activities, creation of training workshops to enhance staff skills, processes for industry co-development and accessing data.

Participants included universities, NHS Trusts and industry, as without their participation, the CHC programme would not be able to achieve its seven deliverables. These organisations have been determined as being separate to the staff inputs (from the inputs section of the logic model) that may come from these organisations. For example, a care pathway may employ a clinician to complete a range of activities; however, an NHS Trust may need to participate as part of the activities being driven by the CHC programme staff to ensure that a data sharing agreement can be used across a range of NHS organisations in one region.

Patients and members of the public were also key participants in the formulation and delivery of some CHC programme activities, such as the citizen’s jury’s and care pathway patient tools. A separate evaluation has been commissioned to fully assess the level of patient and public involvement in the CHC programme.

Outcomes

If the CHC programme has accomplished its planned activities to the extent as planned, then it should have completed or demonstrated progression towards the following seven deliverables: establishment of data sharing strategy and agreements for each region; establishment and delivery of governance arrangements for the sharing and usage of data for each region; workforce arrangements optimised and CPD requirements identified; creation of Arks as analytical platforms; pathway analysis, variation assessment and improvements identifications; frameworks and integration with R&D partners; and the production of a business model suitable for scaling and sustainable for delivery in the NHS.

Impacts

The evaluators were asked by the funders to assess any potential impacts of the CHC programme. Short-term, medium-term and long-term impacts were built into the impact sections of the logic model. Potential short-term impacts include ‘knowledge sharing between organisations’, an ‘iterative cycle of care pathway improvements’ in current CHC programme pathways is achieved and ‘data action latency’ is further developed. Potential medium impacts of the CHC programme include ‘generalisability of CHC approach in other care pathways’, ‘engagement of other organisations’ in the regions to further develop the CHC programme and ‘evaluation of care pathways’. Potential long-term impacts of the CHC programme include ‘tailored approach to local/individual circumstances’, ‘reduction of costs in NHS’ and ‘improvements in patient outcomes’.

Research methods

An evaluation can be described as a systematic process to assess the successes of a programme or intervention and the lessons learnt.12 15 It is based on evaluating a set of activities and formulating a judgement based on the evidence collected to increase the knowledge of programme or intervention for learning; informing the decision-making process for future programmes or interventions and being accountable to stakeholders and donors.15

This evaluation forms a distinct strand within the CHC programme of work, helping to assess progression towards delivery of the regional Arks and each care pathway, rather than a separate study focused solely on the scientific understanding of LHSs. Furthermore, it is important that this evaluation generates evidence to support decision-making within the CHC programme in the future, as well as evidence that assesses progression towards the seven programme deliverables to meet the needs of our funders.

In addition to the central CHC Hub that provides support to the overall programme, each of the CHC regions uses a variety of methods to create their own LHS. Furthermore, each of the eight care pathways within the CHC programme has a different focus with a variety of objectives. Other issue that needed to be taken into account was that at the start of the evaluation, the care pathways were at different points of delivery, with some still in the early development stages and others nearing completion for the first phase of care pathway delivery. Therefore, the data collection method needed to allow for these differences.

In formulating the research design of the evaluation, the following considerations were also adopted: first, the consideration of research ethics to ensure the informed consent and safety of all research participants and the management of confidential data. Second, it was important to ensure that all CHC staff from all regions had an opportunity to provide feedback through the evaluation. Third, to reduce potential interview and survey fatigue, a sufficiently in-depth methodology to meet the evaluation objectives was needed, but light touch where possible to avoid placing an undue burden on participants.

As a result, a mixed methods approach was deemed the most suitable approach to this evaluation. Using a mixed methods approach allows the evaluation to systematically combine and synthesise evidence from the eight care pathways, including a deeper investigation of each care pathway in order to gain a comprehensive understanding of the resources, processes, barriers and facilitators. Furthermore, because baseline data did not exist for all the pathways, using a range of data collection methods would ensure triangulation in order to increase the credibility and validity of the results. The evaluation will centre on three approaches to data collection: a documentary review, semi-structured interviews and an online survey.

Patient and public involvement

While patients and members of the public are involved in the CHC programme, they were not involved in this evaluation. An evaluation solely dedicated to patients and public involvement has been commissioned separately.

Documentary review

A documentary review will be undertaken throughout the duration of the evaluation period. In doing so, we will be able to review pre-existing and new documentation to determine any differences between the proposed CHC pilot study and the actual programme of implementation. In doing so, the documentary review can highlight issues that can be missed through other means of data collection16 and will assist in the formulation of semi-structured interview topic guides and the online survey.

To evaluate progress towards the CHC programme deliverables, documents from different time points in the project will be used to identify the structures and procedures used to deliver each care pathway, as well as the overall CHC programme. This will include monthly project reports, meeting documentation, internal evaluation reports, marketing materials and other project reports.

Online survey for CHC staff

As there are 210 members off staff working on the CHC programme, split across four regions in the north of England, it was felt that conducting an online survey that will be offered to all CHC staff to complete, was a practical approach to ensure all CHC staff had an opportunity to contribute to the evaluation. This is to gain a broad understanding of CHC staff experiences across the different pathways in relation to the CHC programme deliverables. The questions were developed using the logic model and CHC programme deliverables as a guide to ensure questions were relevant to the evaluation. The questionnaire will include the following sections:

Approaches to creating regional learning health systems and pathways.

Challenges experienced and/or managed.

Unintended outputs from being involved on the CHC programme.

Recommendations for facilitating future learning health systems and pathways.

The questionnaire has substantial sections for free text to all staff to describe their experiences in the CHC programme and care pathways in more detail. These sections will be transcribed for qualitative data analysis. A link to the questionnaire will be emailed to all 210 staff across the CHC programme. In addition to the online survey, staff can also request a paper-based copy of the survey, or to complete the survey over the telephone. Data from responses will be exported from the survey handler and securely stored in Microsoft Excel for initial data cleaning and then to SPSS for data analysis.

Semi-structured interviews

We will conduct semi-structured interviews with key CHC staff from all four regions, as well as the central CHC hub. The aim of the interview is to develop a clearer understanding of staff experiences in the design and delivery of the CHC programme and pathways. A topic guide was developed using the logic model and initial results from the documentary review as a framework in which to formulate interview questions. Key areas that would be explored during the interview include the following:

The Learning Healthcare system.

CHC programme deliverables.

Using data in care pathways (such as information governance and data quality).

Patient and public involvement within each region and pathway.

Creating a skilled workforce.

Working with industry.

Using a semi-structured interview methodology would allow the researchers to explore emerging issues during the interview.17 The interviews will take place at the place of work of the participant. All interviews will be audiotaped and transcribed verbatim. Transcribed data will be anonymised to remove any traceable information that could identify the respondent to the transcript (eg, names of people or place names). Each respondent will be assigned a project code and this will be used in place of real names on all collected data. The ‘project key code’ linking project codes to identifiable respondent data will be kept electronically on a password-protected secure server. Digital recordings of interviews will be stored on a password-protected secure server, while hard copies of (anonymised) transcripts and field notes will be kept in a locked filing cabinet, in a locked room.

Analysis

Our data analysis uses a thematic approach where we will triangulate the documentary review, survey and interview data to quantify progress towards the CHC programme deliverables. This is because no baseline data were collected for the CHC programme. Even though it is not possible to determine which pathways will provide data that will allow for a more sophisticated data analysis, where the data allow, we will aim to measure cost reduction and improvements in patient deliverables for each pathway.

Our analysis strategy will also use an iterative process, whereby data collection and data analysis will be conducted concurrently. For data collected through our documentary review and interviews, a thematic analysis using our logic model as a framework will be used to assess progress against the CHC programme deliverables and to identify recommendations to support future programme decision-making. Descriptive analysis of the online survey data will also be used to inform actionable recommendations, which in turn will aid the future development and refinement of the CHC programme and care pathways. Each of the CHC regions will receive an evaluation report to further assist in the regional development of current and future pathways.

Concluding comments

Through this evaluation, a range of evidence will be collated and produced to support a series of evaluation judgements aimed at assessing the seven CHC programme deliverables. This will include a documentary review to identify how CHC programme deliverables were operationalised; an online survey to gain a broad understanding of CHC staff experiences in delivering each pathway and semi-structured interviews with key programme staff to gain a deeper understanding of key achievements and challenges. Using a three-pronged approach ensures triangulation and increases the validity, reliability and credibility of the results.

In planning this evaluation, we have used a logic model to guide the development of the data collection methods. Using a logic model, we have been able to initially identify and set out our short-term, medium-term and long-term impact measures that are linked to the CHC programme deliverables. We do not expect to be able to measure precisely all impact measures due the lack of baseline data, the different pathways in the CHC programme, the different stages of delivery of each pathway and the short time period of the evaluation. However, the data collected will allow us to assess progress made towards the CHC programme deliverables, as well as to determine the types of contributions made and challenges faced for each region in achieving these deliverables. Any future evaluation that considers both the costs of implementation and patient and public involvement, which will assist in determining the feasibility of converting the CHC programme becoming a sustainable model across the UK, is dependent on the funder.

We had to take a pragmatic approach to ensure the feasibility of completing the work within 10 months. Focusing the evaluation on eight care pathways allows for a systematic approach that will give an overview of the key achievements and challenges for each region, as well as the CHC programme overall. In addition, a key output of this evaluation was to assess progress towards the CHC programme deliverables. As a result of this, some aspects may be underexplored. However, as each pathway will be independently evaluated, we are satisfied that this risk has been managed. Thus, in focusing the evaluation on the overall CHC programme deliverables, the evaluation will be grounded on what the programme set out to do. This has the benefit of producing findings and recommendations that can be used in present and future CHC programme decision-making, as well as contributing to the wider discussion of LHSs.