Quality improvement at scale: evaluation of the drivers and barriers to adoption and sustainability of an intervention to reduce late referral in chronic kidney disease.

This quality improvement project aimed to drive large scale and sustained change to reduce the burden of chronic kidney disease in the UK. The intervention is a software program that extracts relevant biochemical data from laboratory databases which then generate graphs of estimated kidney function (eGFR) over time. Graphs showing progressive kidney disease are sent directly back to general practitioners (GPs) to alert them to rereview patient care and if necessary, refer to renal services. The aim of this evaluation study was to explain the barriers and drivers to implementation and adoption of the eGFR graph intervention. This evaluation study involved 5 of the 20 participating renal units (sites) . A developmental evaluation approach was used. Methods included collection of descriptive data about graph reporting; GP surveys (n=68); focus groups (n=4) with practices; face-to-face interviews with secondary care clinicians (n=10). Results showed the mean number of graphs reviewed per week per site was 230, taking 1 hour per week per site. Only 18.2% graphs highlighted a concerning decline in kidney function. Important enablers to sustain the intervention were low cost, easy to understand, a sense of local ownership and perceived impact. Barriers included nephrologists' perceived increase in new referrals. We concluded that developmental evaluation can explain the barriers/drivers to implementation of a national quality improvement project that involves a variety of different stakeholders. The intervention has the potential to slow down progression of kidney disease due to the eGFR prompts alerting GPs to review the patient record and take action, such as reviewing medications and referring to renal teams if progressive kidney disease had not been identified previously. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Chronic kidney disease (CKD) is a major public health issue and stages 3–5 CKD affect 5%–6% of the adult population in the UK.1 If CKD remains undetected and untreated, progression to more advanced stages has a severe impact both on survival and quality of life, especially when renal replacement therapy (RRT) is required.2 Improved identification and management of CKD can be improved through better information systems.3

Referral of patients with CKD from primary to secondary care in the UK is broadly based on the international staging of CKD.4 Patients with CKD stages 1–3 are generally cared for in primary care and referred to secondary care if they reach stage 4.5 The inevitable result of this approach is that many patients with CKD stage 3 that are progressing more quickly towards the more advanced stages of disease, are not referred until relatively late in their progression. Late referral (within 3 months of requiring dialysis) leads to poorer outcomes, such as prolonged hospitalisation, increased mortality, as well as a lower likelihood of planned vascular access or pre-emptive kidney transplantation.6

Aims of the project

A programme to Spread eGFR graph Surveillance for the early Identification, Support and Treatment of people with progressive Chronic Kidney Disease (ASSIST-CKD) aims to drive large scale, measurable and sustainable change that reduces the burden of CKD across the UK, and in particular reduces late presentation for RRT.7 In 2017–2018, the proportion of patients presenting late to secondary care varied across renal units in England, Wales and Northern Ireland from 6.9% to 40.5% (mean 16.1%).8 The recommended preparation time for RRT is 1 year.5

The design of the intervention

The intervention consists of a dedicated graphing software program together with review of estimated glomerular filtration rate (eGFR) graphs by trained staff within secondary care.3 The software extracts relevant biochemical data from hospitals’ pathology laboratory databases and generates graphs of eGFR over time for patients <65 years with an eGFR <50 mL/min/1.73 m2 and patients >65 years with an eGFR <40 mL/min/1.73 m2. Individuals who review the graphs (laboratory staff, a renal pharmacist or a renal nurse) are given virtual training and undertake competency assessment. For those patients whose graphs show a declining eGFR trajectory (as opposed to low but stable function) the laboratory sends the eGFR graph to the patient’s general practitioner (GP) (family doctor) with an alert for them to review kidney function and decide if specialist advice is needed. Some sites have more than one person involved in the intervention: one person undertaking the graph interpretation and another to undertake the administrative tasks such as placing the graphs into envelopes and posting.

The aim is not to mandate referral to secondary care, but simply to highlight individuals that are at high risk and thereby increase opportunities to delay progression and, where this is not possible, to prepare for RRT in a timely fashion. The intervention was previously implemented in one site in 2011 and is considered to be the main driver to reducing late presentation in this site.3 This site now has the lowest late referral rate in the UK (6.9% presenting within 90 days before RRT start).8 The ASSIST-CKD programme aimed to spread this intervention to 20 sites across the UK. This paper reports on one part of the evaluation and was undertaken concurrently as the project progressed. A quantitative evaluation will be reported in early 2021, with the primary outcome being the rate of late presentation, defined as the proportion of patients with end-stage kidney disease first seen by renal services less than 90 days before commencing RRT. This is a key quality metric in kidney disease. Data on late presentation are collected routinely by the UK Renal Registry and will be compared across participating sites before and after implementation of the intervention. Presentation, defined as time between first visit with a nephrologist and start of RRT, will be analysed as a binary outcome, with late presentation defined as time <90 days.

The aim of the mixed-method evaluation reported in this paper is to investigate the barriers and drivers to the adoption and sustainability of the intervention in 5 of the 20 intervention sites. As many successful interventions which are spread to new settings fail to achieve the same impact, or indeed any impact at all,9 it is important to identify the barriers and drivers to adoption and sustainability, in order to facilitate spread to other organisations.

Materials and methods

Developmental evaluation (DE)10 was chosen as the evaluation framework as it is suitable for dynamic, novel and complex environments that are not suited to standard formative and summative evaluation. This intervention is dynamic as there were changes made to the intervention as a result of the local contexts: different countries of the UK (England, Scotland and Wales), IT systems and size of population covered by the renal unit (305 000–600 000 people). In summary, DE can be undertaken where both the path and the destination are evolving.

Unlike traditional evaluation, DE sits inside the project, therefore, it is necessary for the evaluator to be part of the programme development team11 and to recognise, record and feedback unexpected and unpredicted outcomes, in order to facilitate continuous improvement of a project. This happened regularly within this project, during project team and evaluation advisory group (EAG) meetings, and also face-to-face feedback from the evaluator, such as local results of GP survey/focus groups, to individual sites. These 1–1 formal sessions with the evaluator occurred twice during the 2-year programme, although contact with the project team was much more frequent. In addition, the project team ran three learning events, whereby sites shared their learning experiences, alongside open dialogue on best practice methods and review challenges. The purpose was to encourage sites to long-term adoption, sustainability and independence.

DE explores both fidelity and impact of an intervention. The fidelity of the intervention evaluates the degree to which the original intervention has been replicated, which in turn contributes to a viable assessment of its impact.

The first five sites that had implemented the intervention for >1 year (by January 2017) were included in the evaluation. All sites were in England but varied in geographical size (305 000–600 000 population). All sites implemented the intervention as described above, but there were some subtle differences, with one site involving a renal nurse in the interpretation of graphs instead of laboratory staff, and another with a fully automated system whereby graphs were reported electronically to primary care (instead of being sent by mail as was the practice in the other intervention sites).

Patient and public involvement

We recruited a diverse team of ten people to our patient project team (PPT) and overall individuals contributed to the project design and dissemination of the findings, including presentation of their powerful personal and impactful stories at our collaborative learning events for sites; had a major role in developing our business case document and infographic for commissioners and represented the project team at dissemination events. Members of the PPT also contributed to the evaluation measures including development of the GP survey and discussions about whether the showing of eGFR graphs to patients might cause harm.

Evaluation methods and outcomes

Evaluation of fidelity

Fidelity is defined in terms of five elements that need to be measured: adherence, exposure or dose, quality of delivery, participant responsiveness and programme differentiation (elements of the intervention that are essential for its success). Table 1 shows how fidelity was evaluated in each of the five sites.

Table 1

Methods of fidelity capture

Adherence to intervention	Exposure to intervention	Quality of intervention	Responsiveness to intervention	Differentiation elements essential for success
Audit: no of graphs reviewed 3 monthly	Audit: no of graphs reported3 monthly	GP survey12–18 months poststart date	GP surveyReferral rates to secondary care annually	1–1 interviews with nephrologists (project lead in each site) 12–18 months poststart date
1–1 interviews (box 2) with staff who interpret the graphs (laboratory staff or renal nurse) 12–18 months poststart date	Practice focus groups: whether graphs are received by GPs 12–18 months poststart date.	Practice focus groups 12–18 months poststart date4–8 staff, including a practice manager, GP(s) and practice nurse(s).	Practice focus groups 12–18 months poststart date

The GP survey is shown in (online supplemental material interview) questions are shown in box 2.

GP, general practitioner.

Evaluation of impact

Evaluation of impact was carried out between May and July 2018 and comprised either face-to-face or telephone interviews with clinical biochemists/renal nurse and nephrologists in all sites. These interview questions are shown in box 1.

Have you had any specific cases where progressive chronic kidney disease has been stabilised or reversed?

Given the results to date (from the Renal Registry) can you give any insights into these results?

Do you think that the ASSIST-CKD (A programme to Spread eGFR graph Surveillance for the early Identification, Support and Treatment of people with progressive Chronic Kidney Disease) project has impacted on your recent late referral data from the Renal Registry?

What has been better/worse since the project started?

How does/how eGFR surveillance fit in to renal surveillance in your area (new developments)?

Have there been any intended/unintended consequences?

Have you any plans for sustainability, for example, personal factors?

Is there a need for a business case/incentivisation for this to continue?

Have you thought about succession planning?

Is there a need for further continuous professional development (CPD)/training?

Can you explain how you first got involved in the ASSIST-CKD (A programme to Spread eGFR graph Surveillance for the early Identification, Support and Treatment of people with progressive Chronic Kidney Disease) project?

What were the drivers/barriers to getting started?

How has the ASSIST-CKD project been running so far?

What has made it easy/difficult?

Are you seeing any benefits to the intervention so far?

Interviews were taped and transcribed. The interview data were analysed manually using thematic analysis.12 In order to add to the interview data’s trustworthiness,13 another project team member checked and verified the themes and inferences that were emerging.

As with all quality improvement projects, evaluation of sustainability is crucial. Our key marker of sustainability was that a site had implemented eGFR reporting for more than 1 year (ie, reporting for which the sites received no funding or support from the project grant. We check that eGFR data are sent to the UKRR 3 monthly and that new staff are trained and accredited to read eGFR graphs.

Results

Fidelity findings

Adherence

18.2% of reviewed graphs were reported to GPs but there was wide variation (9.3%–28.7%), even when the size of the population was taken into account (see table 2).

Table 2

Numbers of graphs reviewed and reported over a 3-month period

Site	Average no of graphs reviewed over a 3- month period	Average no of graphs sent to GPs over a 3-month period	Average per cent of graphs reviewed that are reported each quarter	Mean no reported each 3 months, per 100 000 population
A	3341.2	1005.6	28.7	222
B	Data not available
C	4703.3	436.5	9.3	109
D	2269.6	482.8	21.2	80
E	1622.8	218.7	13.6	72

Interpretation of the graphs took approximately 1 hour per week per site. The number of graphs reported and sent to GPs was stable over time across all sites, apart from one site which decreased its reporting rate in year 2, due to increasing experience and confidence of the person who reviewed the graphs, so there less reporting of decisions that are borderline (explained at interview).

Exposure

Only a small number of graphs were received by an individual GP practice each month. One site reported 20% of cases where no evidence of the eGFR graph being actioned could be found.

part of the feedback from one or two GPs was that, because it doesn’t say result on it, the receptionist wasn’t passing it on… (Nephrologist)

Quality of intervention

A total of 90% GPs in the survey (out of 68 responses) said that the eGFR graphs were helpful, even though GP management systems in the UK already have the facility to generate eGFR graphs. The additional perceived value offered by ASSIST may be that the ASSIST intervention actively highlights high risk patients to GPs, whereas the graphing facilities within GP software systems only produce graphs on request. There is also a potential benefit for patients as the graphs can be used to support self-management. One GP said:

I show the graphs all the time to patients ……CKD is difficult to explain in simple terms, it is not like a cancer and it is not in the public eye. So, it is actually difficult for them to come to terms with, compared with other illnesses.

Responsivenes

Determining whether GPs act on the graphs is important but can be challenging, particularly when considering the variety of activities carried out within the practice. However, one clinical scientist found that patients who had been flagged often had repeat tests showing improved renal function, implying that some intervention had taken place. Specific examples of where the GPs acted on the graphs included review of kidney function, checking that the patient has been coded for CKD, and making sure that deteriorating kidney function has not been missed.

Differentiation (elements for success)

The ASSIST-CKD software has been continually reviewed and adapted in response to the views of the sites to increase the speed of the software to make reviewing of past results and alerts easier, which has impacted in a positive way on the motivation to continue.

In many sites, it was the setup stage that was critical, with buy-in from the Medical Director of the hospital, the IT Director/manager and IT staff working in the laboratory an essential requirement. Another benefit seen when renal teams report the graphs is that deterioration in patients already known to secondary care is highlighted in advance of, or in between, routine clinic visits. As one renal nurse said

Most weeks I highlight a change in eGFR in one of our patients which can result in them being admitted, or being asked to attend for urgent repeat bloods, to have their out-patients appointment brought forward, or the Consultants sending a letter to their GP regarding medication changes…

In summary, the drivers for adoption are that the intervention can be adapted to changing needs, low cost, quick, easy to understand and clinicians can clearly see the benefit. The barriers are that that there needs to be full buy-in from a variety of stakeholders at the start.

Impact findings

The findings are grouped into five main themes. Table 3 shows the themes and subthemes. Drivers are shown in green and barriers are shown in red.

Table 3

Themes and subthemes related to impact of intervention

Effectiveness	Impacts	Unintended consequences	Outcomes	Sustainability
Lack of understanding by GPs on type of action needed	Action taken by GPs (not referral)	Perceived increase in new referrals to secondary care	Reduction in late referrals (not measured in this part of the evaluation)	Likelihood that the intervention is impacting on late referral
	Enhanced surveillance of patients either not known or known to renal	Quality of information on referrals to secondary care		Positive impact on staffing (continuous professional development (CPD) and reflective practice)
	Graphs providing safety net for nephrologists			Low budget

GP, general practitioner.

Effectiveness

The GP survey identified that GPs were sometimes unclear on what to do following receipt of the graph, with some just referring straight back to secondary care without reviewing the patient. However, the intervention was never intended to give prescriptive advice, but rather to highlight trends to GPs to enable them to use their judgement in the light of their knowledge of the individual patient. Some GPs liked this (as it allows some freedom in decision-making) whereas others did not, preferring directed advice.

Impacts

As one nephrologist has outlined there are multiple benefits for patient care that are not directly captured if the outcome is simply reduction in late referral.

…there are different levels of benefit. So, the GPs just having to review the medication, made a change and that’s made a difference. That’s one: I think that’s probably the best sort of change that we can effect. Then the second is they haven’t but at least they are monitored: that’s a second benefit…the third one is they have looked at it and now they have referred a patient. I think all of that doesn’t get captured if you only look at the end point… (Nephrologist)

In one site, 8/13 GPs indicated that they had done ‘something else’ with 4/8 indicating that they had reviewed medication. It is also possible that graphs are reviewed but this has not resulted in any action because it might not be appropriate, where there are existing conditions such as dementia and frailty. One interviewee commented on the impact on care of those patients already known to the renal team, while a theme that was raised by a number of nephrologists was the eGFR graph acting as reassurance that patients with progressive CKD were not being missed, even if they were referred back to primary care after first review.

Unintended consequences

One unintended consequence of the intervention might have been an increased number of inappropriate referrals. A couple of nephrologists were convinced of the impact ‘we had to put on a lot of extra clinics’, yet others could not be certain. However, new referral data across three of the five sites showed no impact (max 1–2 new referrals per month) following initiation of the ASSIST-CKD intervention.

One site conducted their own local evaluation into referral patterns. Over half the patients who triggered an alert had their blood tests repeated. There were only 20% where no evidence of the alert being actioned (no bloods had been repeated and no referrals made).

As well as the number of referrals that might have been impacted because of the ASSIST-CKD project, another theme which developed during the interviews was the perceived reduction in quality of the information supporting the referral.

I would say the first thing is that the nature of the referral is poor, it has been generated, so the GPs are saying okay you asked for a referral, here is the referral, I won’t give you any details about the patient’s history, so it’s a very short referral letter in general, here is your graph, back to you type of referral. (Nephrologist)

Outcomes

A number of interviewees were very positive about the potential outcomes of the ASSIST-CKD project. The reasons for positivity were anticipated clinical benefit, with the prospect of delaying the need for dialysis is some. However, one interviewee perceived that the benefit outweighed any unintended consequences

I always suspected a hugely increased number of referrals but … if we are preventing ten or even fifteen these crashing (onto) dialysis, that’s probably good. I think—if you look at it—probably preventing some going into that stage. (Clinical biochemist)

The initial possible impact of ASSIST-CKD in this site is extremely encouraging and is driving the site to sustain the intervention for another 2 years, especially as funding has been provided to enable this to happen.

Sustainability

The ASSIST-CKD funding only covered 1-year intervention at each site, but all the evaluation sites continued implementing well beyond this, especially if the system became embedded.

we all felt that it was a good thing to do and we need to do it in a manner where it would not be a huge additional burden to us and it just gets integrated to our routine, clinical work that we do…. it’s very well embedded and, as you know, it’s part of our clinical validation pathway. (Clinical biochemist)

Table 4 shows the sites, start times and reasons for sustaining stopping the intervention.

Table 4

Sustainability of sites

Site	Start date	Finish date	Reason for sustaining/stopping
A	12. Aug.15	Live	Embedded practice into lab weekly work streamLow costObtained support from commissioners to continue projectBelief in initiative - initial outcome data looks encouraging
B	4. Dec.15	30. Oct.17	Inconsistent implementationLoss of data due to in-house IT crashOnly one lab clinical scientist to run intervention, whereas a minimum of two people required to cover sickness and holidays
C	19. Oct.15	Live	Embedded—part of clinical validation pathwayLow costSpread initial funding over 2 yearsBelief in initiative
D	6. Oct.15	31. Jul.18	Lack of admin support to send letters to GPsNew Trust encryption process for software required—not compatible
E	26. Sep.16	Live	Anticipated clinical benefitEmbedded practice into work streamModest staff time required to manage systemFully automated system for reporting results electronically to primary care

GP, general practitioner; IT, Information Technology.

If the evaluation sites are compared with all the participating sites, seven sites have been reporting for more than 3 years, 12 sites for 2 years or more and one site has been active for just 1 year. However, as of 7 October 2020 of the 20 sites have stopped reporting, the reasons given are as follows:- staffing issues in two sites, IT problems in two sites, commissioners withdrew funding in one location and COVID-19 has caused a temporary stop in a further two sites.

Discussion

This is the first study in primary care in the UK to scale up an impactful intervention that concerns identification of progressive CKD, using DE to identify the barriers and drivers. The drivers to implementation and sustainability appear to be the modest time investment (on average 1 hour/week), low implementation costs and stakeholder support. The evaluation has captured the ‘hidden’ work by GPs and highlights the relevance of other actions beyond referral to secondary care. In secondary care, the drivers are enhanced surveillance of patients either not known or known to renal teams, confidence in the ‘safety net’, and a possibility that the intervention is impacting on late referral.

Lincoln and Guba14 propose the concepts of credibility, transferability, dependability and confirmability to address trustworthiness and quality in naturalistic (qualitative) inquiry. Credible findings have been generated because of our peer debriefing sessions (project team meetings), checking our interpretations against raw data (data analysed by the Renal Registry statistician) and member checking (checking interview data with participants). To enhance transferability to other sites we have provided a detailed description of the sites, context and data analysis. Dependability refers to investigators arriving independently at the same or similar interpretations. We ensured robust communication, particularly at critical points, with (external) members of our EAG. Confirmability of findings were assured as data were sourced across various settings and locations. For consistency, one member of the project team (the evaluator) conducted all the interviews and initial coding of the data, but another person undertook detailed coding and cross-checked back with the evaluator. All members of the project team discussed preliminary findings to ensure they were a credible interpretation of participants’ responses.

Fidelity: lessons learnt

We learnt the importance of continual review and adaptation of the software in response to feedback from the sites. The greatest challenge in delivery was technical, as a result of the highly variable IT landscape across the sites. Halfway through the project, in Spring 2016, a radical decision was taken to completely recode the software with a new commercial (third party) developer to make it more efficient, user-friendly and able to support longer-term sustainability. This major upgrade was supported by the Funder and involved a finite 6-month build time and piloting in three National Health Service laboratory test sites to ensure reliable and accurate eGFR graphing results and speed of performance.

Impact: lessons learnt

While the results of the full quantitative evaluation are not yet available, the results of this qualitative evaluation suggests the existence of softer benefits beyond any impact on headline of late referral, which may only become evident after a period of years, for example, review of medication, retesting of eGFR, and discussion of self-management with the patient. Few interventions have focused on identification of those at risk of progressive kidney disease. An American study15 in 11 practices assessed whether clinical decision support could be used to improve CKD identification by using risk assessment tools, health maintenance protocols, flow charts and a patient registry. The authors concluded that barriers such as incorporating use of improvement tools into existing workflow must be addressed to effectively achieve improvements in CKD outcomes. An improvement collaborative in the UK16 with tailored facilitation support appeared to promote the uptake of evidence-based guidance on the identification and management of CKD in primary care. Another UK study17 used trigger tools for identifying CKD from the electronic health record, found that in addition to these tools identifying patients with a falling eGFR, they also prompted review of the eGFR trajectory and management plan, as we found in this evaluation. A systematic review18 identified the most common barrier to identification and management of CKD in primary care was a lack of time, explaining to patients they had CKD and dissatisfaction with CKD guidelines. The most common driver, which resonates with our findings, was supportive technology and also a collaborative relationship between primary and secondary care.

Limitations

We do not have specific details on how GPs responded to the graphs, apart from their responses to the survey and explanations in the focus groups. Further studies in this area are warranted, especially as it is recognised that late referral rates to secondary care can be a very blunt instrument for measuring impact.

Conclusion

The national intervention has proved feasible even without external funding. It has been well received and potentially added value. The evaluation identified four important enablers that sustain a quality improvement intervention: low cost, easy to understand, a sense of local ownership and perceived impact. These enablers should be considered when developing and sustaining any large-scale intervention in primary care. The intervention has the potential to slow down progression of kidney disease due to the eGFR prompts alerting GPs to review the patient record and take action.