Qualitative process evaluation of the Perioperative Quality Improvement Programme (PQIP): study protocol.

INTRODUCTION: The Perioperative Quality Improvement Programme (PQIP) is designed to measure complications after major elective surgery and improve these through feedback of data to clinicians. Previous research suggests that despite the significant resources which go into collecting data for national clinical audits, the information they contain is not always used effectively to improve local services. METHODS AND ANALYSIS: We will conduct a formative process evaluation of PQIP comprising a multisited qualitative study to analyse PQIP's programme theory, barriers, facilitators and wider contextual factors that influence implementation. The research will be carried out with the PQIP project team and six National Health Service (NHS) Trusts in England, selected according to geographical location, type of hospital, size and level of engagement with PQIP. We will include one Trust which has not expressed interest in the PQIP for comparison and to explore the role of secular trend in any changes in practice. We will use semi-structured interviews (up to 144 in Trusts and 12 with the project team), non-participant observations (up to 150  hours) and documentary analysis. We will track the lifecycle of perioperative data, exploring the transformations it undergoes from creation to use. We will use framework analysis with categories both from our research questions and from themes emerging from the data. ETHICS AND DISSEMINATION: Ethical approval has been granted from the University College London Research Ethics Committee (ref 10375/001). Permissions to conduct research at NHS Trusts have been granted by local Research and Development offices in coordination with the Health Research Authority. We will follow guidelines for data security, confidentiality and information governance. Findings will be shared at regular time points with the PQIP project team to inform the implementation of the programme, and with participating NHS Trusts to help them reflect on how they currently use data for improvement of perioperative services. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Multisited qualitative fieldwork will allow us to understand processes of implementation of Perioperative Quality Improvement Programme (PQIP) and its impact, paying close attention to the factors shaping the local context.

Comparison of practice between surgical specialities involved in PQIP and those which are not part of the programme will help differentiate the impact of PQIP from to the changes produced by other factors.

Concurrent data collection and formative feedback as PQIP is being implemented will support rapid iteration of implementation strategies and processes; small numbers of participating sites will limit generalisability of research findings.

The positionality of two of the process evaluation researchers as anaesthetists and as members of the PQIP project team may affect the responses of interviewees.

Introduction

Using data for quality improvement

Improving quality is a priority across healthcare.1 Policymakers are increasingly looking towards systematic methods of using data to improve, rather than simply to assure quality.2–4 However, achieving sustainable quality improvement (QI) has proven to be difficult,5 partly due to challenges such as differing interpretations of ‘quality’6; ineffective data systems5; a lack of informatics or improvement skills among health professionals7; or immature organisational QI leadership.8 These challenges are applicable to the perioperative setting; a recent process evaluation of a data-driven QI initiative in emergency laparotomy care concluded that local contextual factors varied the fidelity of the intervention between organisations.9

Audit and feedback interventions

Audit and feedback (A&F) interventions entail the delivery of performance data or other quality indicators to healthcare professionals over specific periods of time to assist them in identifying areas that require improvement.10 11 The expectation is that healthcare professionals will then use this information to make changes in practice, which will ultimately improve healthcare organisation and the quality of care delivered to patients.12 However, despite significant investment in A&F interventions, there is variability in their effectiveness for improving the quality of care.13 14 Simply providing data to hospitals about their performance may not be sufficient to drive QI.11 13–16 This may partially be due to the lack of theory-based explanations in the design and implementation of A&Finterventions, which leads to a lack of understanding of what an intervention consists of, and what it tries to achieve.11 13 17 In a recent study, Gude et al tried to address this gap by proposing the use of control theory to analyse A&F effectiveness in relation to a series of steps: (1) information–intention (feedback needs to convince recipients of the need to change), (2) intention–behaviour (intentions to change need to be translated into actions) and (3) behaviour–impact (actions need to create actual changes in practice).14 Unfortunately, this study did not explore the third dimension, that is, the extent to which the A&F intervention produced changes in practice.

Another gap in A&F studies is the lack of analysis of the social and organisational context where the A&F intervention is implemented or its process of implementation. Looking at aggregated data to ascertain impacts of outcome-monitoring programmes may not provide a sufficiently detailed understanding of the contexts which promote or impede improvement activity based on quality data.18 Indeed, the success of QI initiatives depends on features of both the initiatives themselves and the contexts in which they are implemented.19 A series of reviews have highlighted that neglected areas of research include the study of fidelity to the model or exposure to the intervention, the mechanisms that formed part of the intervention, staff perceptions of the intervention and the impact of the intervention on behaviour change.10 13 20

The Perioperative Quality Improvement Programme

The Perioperative Quality Improvement Programme (PQIP) is a national initiative to measure complications after major elective surgery and seek to improve these through feedback of data to clinicians.21 22 The PQIP team are aiming to overcome the challenges described above to support clinicians and managers in their use of data using an evidence-based approach. PQIP is led by the National Institute of Academic Anaesthesia Health Services Research Centre, working on behalf of the Royal College of Anaesthetists in collaboration with the Royal Colleges of Surgeons (England), Physicians and Nursing and the Faculties of Intensive Care Medicine and of Pain Medicine and has been adopted to the National Institute of Health Research (NIHR) portfolio of research studies.

The intended activities of PQIP are listed below in table 1. PQIP collects risk-adjusted morbidity and mortality data, as well as processes and patient-reported outcomes for patients undergoing selected elective major operations. Hospitals across the UK began enrolling in December 2016; 90 hospitals are currently enrolled and they have collected data on approximately 12 000 patients. Initially, these data were only collected for six surgical specialties: head and neck, thoracics, upper gastrointestinal (GI), lower GI, hepatobiliary and urology. Subsequently, extra surgical specialties have been added: spinal, complex orthopaedics, burns and plastic surgery. Participating hospitals have the option either to try to recruit all eligible patients or to randomly recruit three to five eligible patients per week using an 8-day rolling sampling cycle. To evaluate potential bias within the sample of patients recruited to PQIP, patient characteristics and outcomes will be compared against those not recruited to PQIP by using an extract of administrative data from Hospital Episode Statistics (HES) held by National Health Service (NHS) Digital. Patient-level PQIP data will also be linked with HES and the Office of National Statistics mortality register in order to track readmission and long-term mortality rates. PQIP will also consider patient-level linkage with other relevant registries and National Clinical Audits in order to provide a comprehensive dataset at the lowest local data collection burden. Patients must give informed written consent to participate in the study, in order that identifiable information can be gathered to permit such linkage with external datasets.

Table 1

PQIP activities

Activity	Description
Creation of network of local collaborators	Local leads have been identified at each site for surgery, anaesthesia, management and improvement. These local leads receive regular newsletters and will be invited to collaborative meetings.
Collection of local data to populate national dataset	The dataset was agreed with multidisciplinary consensus by a Clinical Reference Group. Local sites choose pragmatic recruitment strategies and organise data collection/entry according to local resources.
Feedback of timely data to local collaborators	Near real-time data are displayed for selected metrics as dashboard graphics. All data are (anonymously) available for download by local collaborators. Quarterly summaries and an annual report will be distributed only a few days after the end of data collection for the relevant periods.
Support for multimodal dissemination of data to local collaborators	Local data will be used to create one-page infographics, PowerPoint presentations and PDF reports on a quarterly basis.
Facilitation of local QI	Online training resources will be collated and advertised via the PQIP website and App. Examples of high-achieving hospitals will be shared among participating sites in order to share best practice.
Formative evaluation of PQIP	This process evaluation will contribute to ongoing reflection regarding the successes and challenges of PQIP.
Facilitation of national or local research	The aggregated national dataset will be analysed by the project team (and other researchers) to answer observational research questions. The dataset and webtool will also be flexible to allow its secondary use for other parallel studies aiming for efficient trial designs.

PQIP, perioperative quality improvement programme; QI, quality improvement.

PQIP has set up a website with an online database for data entry and, through this, hospitals receive near real-time feedback of their results via dashboards, along with quarterly and yearly reports. Local collaborators will also be able to download anonymised versions of their entire local PQIP dataset, comprising patient characteristics, perioperative processes, clinical outcomes and patient-related outcome measures. These hospital-level data and dashboards, added to the QI tools from the website, are intended to support local and national improvement in the perioperative care of patients. The expectation is that PQIP will provide a significant patient dataset for research, which can be used both locally and nationally, to inform improvements in patient care and satisfaction.

The aims of PQIP are to:

Reduce the burden of perioperative complications.

Support local QI.

Facilitate national and local research.

PQIP hopes to provide local collaborators with sufficiently useful data to support QI of structures or processes related to the incidence or severity of perioperative complications. This process evaluation will further elaborate the programme theories, as understood by either central or local collaborators, as to how this improvement might, does or does not happen.

Aims and objectives

Aim

The aim of this study is to carry out a formative process evaluation of PQIP by focusing on the programme theories guiding the programme and processes of implementation.

Document the processes used to design the programme.

Examine the views of stakeholders on the programme and how these change through different stages of implementation.

Analyse how the programme is implemented.

Explore the impact of the programme.

What are the programme theories (central and at local sites) guiding the design and implementation of PQIP and how do these change through time?

What are the factors acting as barriers and enablers in the implementation of PQIP?

How did specific contexts shape PQIP and processes of implementation?

What was the impact of the programme on professional cultures and how data are used for improvement?

What are the limitations of PQIP and how can the programme be improved?

In the case of non-PQIP hospitals, how are data normally used (or not)? What are the perceptions of PQIP?

How does the use of data for improvement compare between surgical specialties included in PQIP and those which are not part of the programme?

Study design

The study has been designed as a formative process evaluation based on multisited qualitative research. Previous research on QI interventions has indicated that evaluations of complex interventions should not only assess health and organisational outcomes at the end of implementation, they should also seek to understand the intervention design and utilisation plan, implementers’ underlying ideas about its purpose and effectiveness, how specific contexts have shaped the intervention and process of implementation, mechanisms used to integrate or assimilate the intervention into routine practice, and the degree of exposure to the intervention experienced by implementers and target groups.23–25 Furthermore, when evaluations are formative, study findings can be used to guide implementation efforts in the attempt to produce positive outcomes.26

The study will explore aspects of the subjective qualities of data used in improvement.27 According to these approaches, data are never raw, but undergo a process of contextualisation that requires framing and interpretation.28 Interpretations of data are also subject to negotiation by various clinical or non-clinical stakeholders. These approaches will represent the lens through which we explore the creation, packaging, dissemination and ‘use’ of data for QI within the PQIP.

The evaluation will take into consideration three main dimensions of the intervention:

Individual and organisational ideas of change (motivations, attitudes, social climate).

Practices involved in the design and implementation of the programme (decision-making models, formal and informal behaviour, actions taken to incorporate the programme into practice).

Contextualisation (the interaction of the programme with other local and national initiatives, variability in the implementation of the programme across the organisation).

In order to explore these dimensions, the process evaluation multisited qualitative fieldwork will include two components:

Analysis of PQIP designers’ and implementers’ (PQIP leads and PQIP central study team) perceptions and daily practices to understand:

PQIP programme theories.

PQIP processes of implementation.

Multisited fieldwork in six NHS Trusts (five engaging in PQIP and one not engaging with PQIP) using two waves of data collection to explore:

Local programme theories.

Expectations of PQIP.

Experiences with PQIP.

The extent to which PQIP is embedded in daily practice.

The formative design will entail working with local and central teams to set up regular feedback mechanisms to help shape the programme (and Trusts’ responses to it) across its different stages. This will involve regular feedback to the PQIP project team during their meetings and to participating NHS Trustsafter each wave of data collection; dissemination of findings to wider clinical audiences including oral presentations at national meetings or articles in professional publications.

Two members of the research team are both anaesthetists and members of the PQIP project team (SRM is the project lead). In order to maintain critical distance of the research team, we have involved two other researchers (CV and NF) who are full-time university academics who are not involved in the implementation of PQIP. We will explicitly account for the subjectivity of the research team by engaging in continuous processes of reflexivity, looking for evidence of how researchers/participants acknowledge and respond to the positions of the researchers, and by triangulating between settings when researchers are present and when they are not (for example by using minutes of meetings which researchers have not observed).

Data collection

PQIP project team

We will carry out semi-structured interviews and observations with PQIP project team members. The interviews will focus on team members’ understanding of PQIP’s programme theory, their contributions to the planning and implementation of the programme, factors that have acted as barriers and enablers and plans for the future. An interview topic guide will be developed in order to ensure consistency across different participants. Data collection will take place in two waves: early in the evaluation (2–3 months from the start of data collection) and at a later stage in the evaluation (18 months after the start of data collection).

Observations will be carried out during formal working group meetings and other relevant meetings identified by discussion with the project team. The researchers will also observe and document discussions staff members have in relation to data analysis and dissemination. The observations will be used to understand the processes involved in different stages of implementation. A structured observation guide will be used to enable a consistent approach to recording when decisions are made, who is involved, and the decision-making processes used.

We will collect documents related to the programme design produced by working group members. We will analyse these documents to understand how PQIP, and its implementation, is explained and presented. This documentary analysis will also allow us to track changes in the programme over time. The documents will include meeting minutes, business plans, reports and working papers.

Fieldwork will be carried out by a full-time qualitative researcher (CV). The positions of the researchers will be declared to all participants and consistently reflected on during data collection and analysis.

Multisited fieldwork

We will combine interviews and observations with NHS staff at six trusts, five of which are participating in PQIP and one of which is not. We will also carry out documentary analysis. The fieldwork will be organised in two waves (1–4 months from the start of data collection and 12–15 months after the start of data collection) to capture changes over time.

The interviews will be semi-structured and use open-ended questions. All interviews will be audio recorded and an interview topic guide will be used to ensure consistency across different sites. The interviews will focus on staff members’ motivations and attitudes regarding the use of data to make improvements as well as their expectations and experiences of PQIP, where appropriate. It will also explore their perceptions of how the local social climate affects the use of data.

Observations will be carried out during local meetings and other relevant events (ie, team meetings, department meetings, PQIP meetings, board and quality/safety meetings). The observations will be used to understand staff members’ interaction with PQIP and document any contextual factors that influence whether and how data are used for QI. A structured observation guide will be used to enable a consistent approach to recording field notes across different sites and researchers. This guide will be iteratively updated to explore themes emerging from the data. Documents produced locally will be collected to explore how data are used for improvement, mechanisms set in place to participate in PQIP and wider contextual issues that might influence the ways in which local sites participate in PQIP.

Fieldwork in two hospitals will be carried out by one researcher (DW) as part of a PhD thesis; he is an academic anaesthetist and member of the PQIP team. Fieldwork in the other four hospitals will be carried out by a full-time qualitative researcher (CV). The positions of the researchers will be declared to all participants and consistently reflected on during data collection and analysis.

Sampling

PQIP project team participants

Interviews will be carried out with three main groups: PQIP lead, project team members and PQIP fellows (see table 2). These participants will be asked to take part in two interviews, the first one at an early stage in the evaluation (2–3 months) and the second one mid-way throughout the evaluation (18 months). The observations will be carried out during regular group and team meetings.

Table 2

Sampling strategy for interviews with designers and implementers and meeting observations

	First interview	Second interview	Total interviews	Observations
PQIP lead	1	1	2
PQIP project team	3	3	6
PQIP fellows	2	2	4
Total	6	6	12	~20 hours

PQIP, perioperative quality improvement programme.

Observations (~20 hours) with the PQIP project team will include project team meetings, Clinical Reference Group meetings and other relevant working group meetings at the Royal College of Anaesthetists.

Surgical specialties

We will focus the multisited fieldwork on a single PQIP surgical specialty, lower GI, as this is widely performed, and therefore offers greater variety when exploring context. Orthopaedic surgery (not involved in PQIP) will be studied in comparison to lower GI. Orthopaedics has been chosen because it is performed in a similarly broad range of hospitals to lower GI surgery, and both specialties have established National Clinical Audit data for both elective and emergency surgical services. In addition, a recent initiative called Getting It Right First Time (‘GIRFT’) has been piloted in orthopaedics. The GIRFT team aim to help hospitals improve quality by using site visits to support targeted self-assessment and peer review of local data, with the emphasis on reducing variation between hospitals. This approach for using data has gained traction within the Department of Health and is therefore a timely comparison to PQIP’s approach, which has the same aim but different methods.

NHS Trusts

The process evaluation aims to build a rich picture of how the context(s) of individual hospitals influence PQIP’s impact. Participating hospitals will be NHS hospitals performing both lower GI and orthopaedic surgery and planning to start, or have recently started, recruiting patients to PQIP. One non-PQIP hospital will also be recruited to enable some comparison to secular trends. The same methods for data collection and analysis will be used at all sites.

Six hospitals have been purposively recruited (table 3) to achieve diversity according to the following criteria:

Table 3

Sampling strategy for hospitals

	Site A	Site B	Site C	Site D	Site E	Site F
Location	South East (urban)	London	South East (rural)	Herefordshire and parts of Wales (rural)	Yorkshire (urban)	Midlands (urban)
Academic status	Teaching hospital	DGH	DGH	DGH	Teaching hospital	Teaching hospital
Bed count (small<500 beds<large)	Large	Small	Small	Small	Large	Large
PQIP engagement	High	Low	None	Low	Medium	High
Length of time in PQIP	Less than 1 year	Less than 1 year	None	Less than 1 year	18 months	Over 2 years

DGH, District General Hospital; PQIP, perioperative quality improvement programme.

Geographical location (eg, variation across the country and balance of urban and rural).

Academic status (ie, teaching hospitals vs District General Hospital).

Size (eg, whether they have more or fewer than 500 beds).

Engagement with PQIP (high to low level of patient recruitment for PQIP and non-PQIP site).

Length of time participating in PQIP.

NHS staff participants

Semi-structured Interviews

Approximately 12 staff members will be interviewed per wave of data collection in six sites (n=144). Purposive sampling will be undertaken within organisations using a ‘vertical slice’ methodology (see table 4), where staff from different organisational layers are recruited to capture a wide range of views. Furthermore, horizontal selection of participants will be carried out to ensure recruitment across different professional groups. Stakeholders to be interviewed in each hospital will include clinical leads responsible for perioperative services, PQIP local leads, non-clinical managers, consultants in surgery and anaesthesia, trainees in surgery and anaesthesia, research nurses, other nursing staff and allied health professionals. We will also use snowball sampling to approach potential participants recommended by our interviewees.

Table 4

Sampling strategy for interviews at NHS sites

Wave	Site A		Site B		Site C		Site D		Site E		Site F		Total
	First	Second	First	Second	First	Second	First	Second	First	Second	First	Second
Clinical or medical director responsible for perioperative services	1	1	1	1	1	1	1	1	1	1	1	1	12
Local PQIP leads (surgery, anaesthesia, management, improvement)	4	4	4	4	4 (leads for these areas as no PQIP leads)	4	4	4	4	4	4	4	48
Consultants (surgery)	1	1	1	1	1	1	1	1	1	1	1	1	12
Consultants (anaesthesia)	1	1	1	1	1	1	1	1	1	1	1	1	12
Trainees (surgery)	1	1	1	1	1	1	1	1	1	1	1	1	12
Trainees (anaesthesia)	1	1	1	1	1	1	1	1	1	1	1	1	12
Research nurses	1	1	1	1	1	1	1	1	1	1	1	1	12
Nursing staff involved in perioperative care	1	1	1	1	1	1	1	1	1	1	1	1	12
Allied health professoinals	1	1	1	1	1	1	1	1	1	1	1	1	12
Total	12	12	12	12	12	12	12	12	12	12	12	12	144

PQIP, perioperative quality improvement programme

Non-participant observation

Relevant settings for observation will be identified with local PQIP leads. We will carry out up to 130 hours of observation across the six sites. Settings for observations will likely include training meetings and hospital fora where data are discussed such as departmental, audit, education, morbidity and mortality, or hospital board meetings.

Recruitment

Interviews

The participant information sheet (PIS) will be provided to all potential interviewees at least 48 hours in advance of any planned interview. Potential participants will then be given the opportunity to discuss the PIS with the researcher and ask questions. The researchers will discuss the consent form with interviewees before requesting written informed consent from them prior to interviews. Staff will be able to withdraw consent at any time before or during interviews. In the event of consent being withdrawn after the completion of an interview, the data provided prior to withdrawal will be retained (anonymised fully) for analysis and publication.

Observations

Before attending any specific meetings, the researcher will contact and seek permission to attend from the meeting chair. The PIS will be circulated to all members with the meeting papers; this will reiterate the study details and the opportunity for members to opt out of the observation. The PIS will also be available at the beginning of the meeting. If a member declines consent, their contributions to the meeting will not be recorded in the field notes, or the researcher will withdraw from the meeting if appropriate.

For the observations, staff interacting with PQIP data will be initially approached by the researcher who will provide them with a copy of the PIS. The researcher will give the staff member a brief description of the evaluation aims, what participation entails, and the possibility of not being included in the observations. Verbal consent of the participants will be recorded. We will give staff the opportunity of not having their activities recorded in our observations and assure them that the researchers have previous experience working with this method and will withdraw from situations where it is not appropriate for observations to be carried out.

Data analysis

Transcripts and field notes will be imported into NVivo (QSR International V.11, 2017) and analysed using framework analysis to address the thematic categories listed above which relate to our research questions, and also exploring additional categories which emerge from the growing dataset. A codebook will be developed to facilitate team-based analysis. The same codebook will be applied to the analysis of observation interview and documentary data, thus allowing the cross-referencing of data.

Data gathered from PQIP project team will be used to reconstruct the programme theory as well as understand the history of the programme and its implementation processes. Data gathered from NHS staff will identify their local programme theories, experiences with PQIP, processes of implementation and how the programme interacts with local contexts. Data will be compared between different Trusts participating in PQIP, between Trusts participating in PQIP and one which is not, between varying clinical and non-clinical professions and between two surgical specialties (orthopaedics and lower GI).

Using the interviews, observations and documents described in the previous sections, we will be interested in tracking the lifecycle of the data, from creation to use, paying close attention to transformations suffered throughout the process.29 30 We will map these steps across organisational boundaries, following the flow of data from local sites, to the PQIP team, back to local sites and to other areas where data might be sent or shared. We will be particularly interested in these aspects:

Variability in perceptions of data and ways in which perceptions influence data collection and use.

Preferences for the analysis, display and presentation of data.

Identification of aspects of data amenable to local QI.

Reflexivity

The researchers will engage in a continuous process of reflexivity by documenting their own (and each other’s) assumptions, viewpoints and impacts at research sites. We will also seek any differences in participants’ behaviour between settings when researchers are present and when they are absent (for example by by comparing observations notes with minutes of meetings which researchers have not observed).

Patient and public involvement

We are grateful for feedback on an early draft of our protocol from lay members of the PQIP project team. We will continue to seek their advice regarding data collection, interpretation and dissemination of findings.

Ethics and regulatory approvals

The study has been sponsored by the Joint Research Office at UCL/UCLH (reference number 17/0095). Local permissions to conduct research at NHS sites have been granted local Research and Development offices in coordination with the Health Research Authority (IRAS 222320).

Publication and dissemination policy

Findings will be formatively shared with the PQIP project team during their meetings to allow evolution of their processes. Findings will also be shared with local sites after each wave of data collection to help them reflect on how they currently use data for improvement of perioperative services. In addition, manuscripts detailing the quantitative and qualitative evaluations will be submitted to peer-reviewed journals for publication. Presentations will be prepared for professional and lay audiences.