The National Endoscopy Database (NED) Automated Performance Reports to Improve Quality Outcomes Trial (APRIQOT) randomized controlled trial design.

Background and study aims  Colonoscopists with low polyp detection have higher post colonoscopy colorectal cancer incidence and mortality rates. The United Kingdom's National Endoscopy Database (NED) automatically captures patient level data in real time and provides endoscopy key performance indicators (KPI) at a national, endoscopy center, and individual level. Using an electronic behavior change intervention, the primary objective of this study is to assess if automated feedback of endoscopist and endoscopy center-level optimal procedure-adjusted detection KPI (opadKPI) improves polyp detection performance. Methods  This multicenter, prospective, cluster-randomized controlled trial is randomizing NHS endoscopy centres to either intervention or control. The intervention is targeted at independent colonoscopists and each center's endoscopy lead. The intervention reports are evidence-based from endoscopist qualitative interviews and informed by psychological theories of behavior. NED automatically creates monthly reports providing an opadKPI, using mean number of polyps, and an action plan. The primary outcome is opadKPI comparing endoscopists in intervention and control centers at 9 months. Secondary outcomes include other KPI and proximal detection measures at 9 and 12 months. A nested histological validation study will correlate opadKPI to adenoma detection rate at the center level. A cost-effectiveness and budget impact analysis will be undertaken. Conclusion  If the intervention is efficacious and cost-effective, we will showcase the potential of this learning health system, which can be implemented at local and national levels to improve colonoscopy quality, and demonstrate that an automated system that collects, analyses, and disseminates real-time clinical data can deliver evidence- and theory-informed feedback. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commecial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

RCT Entities:   Population Interventions Outcomes

Introduction

Colorectal cancer (CRC) is diagnosed in half a million people each year in Europe, and is the fourth most common cancer in the UK. CRCs arise from polyps, and polyp detection and resection at colonoscopy is pivotal in preventing CRC. While most CRCs arise from adenomatous polyps, the serrated CRC pathway is now thought to account for up to 30 % of CRCs. Serrated polyps are often subtle, difficult to detect and usually occur in the proximal colon, which might explain why proximal CRCs are missed more thanr twice as often as distal CRCs. The diagnosis of CRC between 6 and 36 months after a cancer-negative colonoscopy is called post-colonoscopy colorectal cancer (PCCRC). Colonoscopists with a low polyp detection rate have higher PCCRC incidence and mortality rates, therefore, people die from unwarranted variation in colonoscopy quality.

International studies show a significant variation in quality markers of polyp detection and withdrawal time between endoscopists. In the UK over 900,000 colonoscopies are performed each year. The current extent of variation in endoscopy quality in the UK is unknown; most data derives from ad-hoc local audits which are often small, use non-standardized approaches, and lack generalisability. This makes identifying underperformance and intervening to improve this difficult.

The UK is the first country to develop a National Endoscopy Database (NED), a novel registry that captures patient-level data automatically and in real time from clinical data entered into each hospital’s endoscopy reporting system (ERS). NED began April 2019 and roll out is almost complete, with over 400 UK endoscopy centers enrolled.

It has been consistently demonstrated that providing clinicians with feedback on their performance is modestly effective at changing behavior and improving performance; a Cochrane review of the effect of feedback using a dichotomous outcome on compliance with desired practice demonstrated a median improvement in practice of 4.3 %. Previous audit and feedback intervention studies in colonoscopy have heterogenous results, but suggest feedback may be valuable, these studies have all had limitations including quasi-experimental design and small numbers of centers or endoscopists. An American longitudinal cohort study of 16 gastroenterologists in a bowel screening program demonstrated an increase in performance in adenoma detection rate with monthly over quarterly feedback. Most feedback interventions in endoscopy and other settings did not have behavioral theories underpinning their design to facilitate understanding of how they change behavior and maximize intervention impact. More generally, there is a sparsity of evidence for using health informatics systems such as NED to provide theory-informed feedback.

For the first time in the context of endoscopy, the NED-APRIQOT study is testing the efficacy of recommending evidence-based behaviors to improve polyp detection using an intervention designed with behavior change theories. We hypothesize that our intervention – monthly, entirely automated feedback of endoscopist and endoscopy center level performance data through NED, of optimal procedure-adjusted detection key performance indicator (opadKPI) – will improve performance and reduce unwarranted variation in colonoscopy polyp detection performance.

Methods

Study design

NED-APRIQOT is a multicenter, prospective, cluster-randomized controlled trial (RCT) which randomizes NHS endoscopy centers into either intervention or control ( Fig. 1 ). The intervention is targeted at independent colonoscopists and the clinician with the role of endoscopy lead in each center.

Fig. 1

 CONSORT extension for cluster trials diagram.

To be eligible, centers must be in England, perform at least 600 colonoscopies per annum and have been data-uploading to NED since August 2019. Participants (colonoscopists) will be recruited within eligible centers that agree to participate. in the trial. All consenting independently practicing colonoscopists employed by and practicing within the trial endoscopy center will be included. Colonoscopists in non-independent (supervised) training will be excluded. In all trial centers, all colonoscopists who fulfill the inclusion criteria will be provided with the participant information leaflet by email. Consent from colonoscopists will be obtained by the local Principal Investigator or a member of the research team nominated on the delegation-log.

The trial will run for 9 months providing feedback monthly, as detailed below. Outcomes will be analyzed at 9 months (primary outcome) and at 12 months (to assess whether there is a sustained response post-intervention).

Optimal procedure-adjusted detection key performance indicator (opadKPI)

The traditional colonoscopy detection key performance indicator (KPI) of adenoma detection rate (ADR) is dependent on histological data. Such data is not available through NED. In addition, ADR has been criticized for the “one-and-done” phenomenon therefore not encouraging multiple polyp detection. ADR does not include non-adenomatous polyps 17 and its binary nature means variation in performance between procedures is limited.

Recognizing unwarranted variation in a KPI is challenging, and it is accepted that case-mix has a significant impact on measures of performance 18 19 It is possible to compute a KPI adjusted for case-mix if variables such as patient age, sex and indication are available, however this is not currently done routinely 20 .

Prior to the study we undertook a scoping literature review to identify candidate KPIs for polyp detection in colonoscopy. We undertook a Delphi process of expert endoscopists to assess which candidate KPIs are likely to be acceptable to the endoscopist population. We also analyzed 100,000 colonoscopies from NED to investigate the variability in a series of candidate KPIs and the factors that influence this. We explored whether case-mix adjusted candidate KPIs can be computed and whether it is possible to do this routinely within NED. This work will be published elsewhere, however, the selected adjusted KPI which is acceptable, discriminated by endoscopists, and correlated well to other performance measures was the mean number of polyps (MNP). The MNP is our optimal procedure-adjusted detection KPI (opadKPI) and will be the primary outcome for the trial and expressed in terms of MNP per 100 colonoscopies.

Nested detection histological validation study

To assess the correlation between ADR, serrated polyp detection rate and opadKPI; in trial centers research teams will be asked to review the endoscopy report and histology data for all independent colonoscopies undertaken over a selected two-week period. For each procedure they will collect data about polyp location, size, morphology, management plan, and histological findings.

Intervention design

The feedback intervention will be both evidence-based, in recommending behaviors associated with polyp detection, and informed by psychological theories of behavior change. A prototype feedback intervention (a personal performance report for individual endoscopists and a center performance report for endoscopy leads) was designed using the theory of planned behavior 21 , feedback intervention theory 22 , and following guidance on the rigorous and systematic development of complex interventions 23 24 .

Prior to the trial, a series of semi-structured interviews were conducted with endoscopists across the UK. These interviews explored (1) endoscopists’ views on feedback, what influences performance, and what influences changes in performance and (2) endoscopists’ reactions to the prototype intervention report. Several rounds of interviews (approximately 4 per round) were undertaken with the prototype intervention refined, revised and then explored in the next round.

The feedback intervention is a monthly email which will be produced automatically through NED. This will show the headline opadKPI figure for that endoscopist, with a social comparison, a target and aspirational standard on a visual scale.

Endoscopists are strongly motivated by peer comparison, in line with behavioral psychology literature on declarative norms 21 25 , and interact more with a colored graphic representation of data 12 . In keeping with the Cochrane review’s findings, an action plan was provided reviewing secondary performance behaviors of proximal detection, withdrawal time, hyoscine butylbromide prescription and rectal retroversion based on UK performance quality recommendations 10 26 . The report includes monthly tips and detailed information of the last 4 months’ performance, with a trend review. All previous reports, action plans and educational material to improve detection will be made available through click links to a bespoke NED-APRIQOT website. Recipients will be provided with contact details of their endoscopy lead to flag concerns about their performance. A similar report has been developed for endoscopy leads. This will include endoscopy center level detection performance, an action plan, a summary of each enrolled endoscopist’s opadKPI and their action plan. The endoscopy lead has responsibility for quality assurance within their center, and, as recommended by JAG, is encouraged to support all endoscopists, particularly those demonstrating potential underperformance 27 . All recipients will be provided with contact details of local investigators and the central research team to flag any perceived discrepancy in the accuracy of the feedback data.

Control arm and usual audit and feedback

The control centers and intervention centers will continue to follow their usual practice with respect to regular audit and feedback processes. In brief, JAG recommends sites provide regular (6-month) feedback on each endoscopist’s KPIs and discussion at the endoscopy center’s user group meetings. The KPI reviewed includes cecal intubation rate, ADR, bowel preparation, withdrawal times, sedation use, number of procedures performed, comfort level and center level post-colonoscopy colorectal cancer 10 27 28 . Whether sites follow recommended processes will not be monitored in the trial.

Endoscopists at control centers will have access to the current NED website, providing information on basic KPI data 11 , however, they will not actively receive data from NED or have access to the enhanced NED-APRIQOT website.

Sample size and randomization

We anticipate recruiting all independent endoscopists from up to 50 endoscopy centers across the UK. All colonoscopies conducted by those endoscopists irrespective of indication, bowel preparation quality or prior colonic surgery will be included in the data analysis, apart from training procedures and incomplete colonoscopies (where the caecum/ileum is not reached).

A simulation technique was used for sample size estimation based on a multilevel Poisson model. Estimating relevant parameters from existing data resulted in at least 80 % power to detect a minimum important improvement of seven polyps detected per 100 colonoscopies between intervention and control arms assuming 5 % alpha (two-sided test), 50 sites with 20 endoscopists per site and 15 colonoscopies per endoscopist per site per month.

The unit of randomization is the endoscopy center, where “endoscopy center” is defined as an endoscopy provider (usually a hospital Trust) which might comprise more than one site but which has a single workforce of endoscopists and central audit and feedback processes across those sites. Randomization will be performed in a 1:1 ratio to intervention and control arms, using a stratified randomization scheme to account for Trust workflow and baseline polyp detection rate in the preceding quarter, as these are likely to be major determinants of the outcome.

Primary and secondary outcomes

The primary objective for this clustered RCT is the opadKPI comparing endoscopists in centers receiving regular feedback and endoscopists in the control group, measured at 9 months.

The secondary outcomes comparing baseline data to post-intervention data at 9 and 12 months, measured in both arms and compared are:

% endoscopists above minimum standards for the opadKPI

% endoscopists above target standards for the opadKPI

mean value of procedure-adjusted proximal polypectomy rate (PPR)

% endoscopists above minimum standard for procedure-adjusted PPR

% endoscopists above target standard for procedure-adjusted PPR

change over time in the opadKPI

change over time in procedure-adjusted PPR

mean value in other colonoscopy KPIs (cecal intubation rate, terminal ileal intubation rate, rectal retroversion rate, colonoscopy withdrawal time, polyp retrieval success, patient comfort, bowel preparation quality)

We will use a minimum relative standard of the 25 th percentile and aspirational target standard of the 75 th percentile at the start of the study for opadKPI and PPR. We will also analyse an absolute minimum standard pf 80 polyps per 100 procedures for MNP, as demonstrated in a bowel screening population, with an equivalence of 35 % ADR 29 .

As part of a nested detection histological validation study, a further secondary outcome will be a center level correlation between opadKPI and both ADR and serrated polyp detection rate for a 2-week collection period.

We will also compare opadKPI at the center level and endoscopist level with other quality metrics, namely:

Endoscopy center-level (the NED-APRIQOT team will obtain these from national datasets)

JAG Global Rating Scale clinical quality domain/individual components of this domain

JAG accreditation status

Whether the center is:

a Bowel Cancer Screening Program screening center

JAG regional training center

PCCRC rate

Endoscopist level:

Annual workload

Lifetime colonoscopy experience

Number of years independent practice

Screening colonoscopy accreditation status

JAG “train the trainers” accreditation status

Specialty

All data that will be collected during the study is summarized in Table 1 .

Data collected during the study at 9 and 12 months.

Anonymized procedural data	Patient data	Age
		Sex
		Procedure indication
		Diagnosis
	List data	Site of endoscopy list
		Date and AM or PM list
		Other procedures undertaken on the list
	Procedural data	Extent of examination
		Magnetic imager/ScopeGuide used
		Rectal retroversion
		Digital rectal examination performed
		Bowel preparation score
		Patient comfort score
		Medications prescribed during the procedure
		Adverse events during procedure
	Detection data	Number of polyps identified
		Any polypectomy performed
		Polypectomy technique
		Location of polypectomy
		Size of polyp ( < 1 cm, 1 – 2 cm, > 2 cm)
		Polyp retrieval status
Endoscopist data	Baseline information	Age
		Gender
		Background specialty
		Annual workload
		Lifetime colonoscopy experience
		Number of years’ independent practice
		Screening colonoscopy accreditation status
		JAG “train the trainers” accreditation status
	Endoscopist detection data	Mean number of polyps per 100 procedures adjusted for age, sex and indication
		Mean number of polyps per 100 procedures
		PPR adjusted for age, sex and indication.
		PPR
		Polyp detection rate
		Mean withdrawal time excluding procedures performing therapeutics.
		Percentage buscopan prescription
		Percentage rectal retroversion rate
	Endoscopist interaction data	Frequency of access to NED APRIQOT and NED websites
		Pages visited on the NED APRIQOT and NED websites and for each page length of time spent, and date and time of each visit.
		Device type used to access NED APRIQOT and NED websites (work computer, home computer, tablet, mobile device).
Endoscopy center data	Center detection data	Mean number of polyps per 100 procedures adjusted for age, sex and indication
		Mean number of polyps per 100 procedures
		PPR adjusted for age, sex and indication.
		PPR
		Polyp detection rate
		Mean withdrawal time excluding procedures performing therapeutics.
		Percentage buscopan prescription
		Percentage rectal retroversion rate
	Histology validation data, for 2-week period	Polyp detection rate
		ADR
		Serrated polyp detection rate
	Health economics analysis	Baseline survey of endoscopy center costs

NED, national endoscopy database; APRIQOT, Automated Performance Reports to Improve Quality Outcomes Trial; PPR, proximal polypectomy rate; ADR, adenoma detection rate.

Data analysis

Primary analysis will follow intention-to-treat principles with endoscopy centers analyzed according to randomization arm and irrespective of whether they received (or acted upon) the intervention; other analysis approaches such as per-protocol may be considered subsequently. No interim analysis is planned.

Analysis of mean difference in opadKPI between the trial arms at 9 months and 12 months post-randomization will be analyzed using generalized estimating equations for non-Gaussian data and with endoscopy centers as a clustering variable to account for intra-center correlation.

Longitudinal analysis techniques, using generalized estimating equations, will be used to investigate changes in opadKPI and procedure-adjusted PPR from baseline to 9 and 12 months using generalized estimating equations with individual endoscopists as clustering variable to account for repeated data per endoscopist. This model will also incorporate a second random effect for endoscopists nested within centers to account for intra-center correlation. The repeated time points (baseline to 9 months) will be specified as scores if the associations in the changes in outcomes are linear over time, otherwise the time points will be specified as categorical variables with baseline a reference group. The model will also investigate whether variation between endoscopists is constant over time by comparing a random intercept and random slope models. A likelihood ratio test statistic for variance components will be used to compare the models.

There will also be subgroup analysis of endoscopists in the intervention group who access learning resource materials on the NED APRIQOT website during the trial, along with endoscopists in the control group who access the basic NED website during the trial. Descriptive statistics by trial arms including other KPI will be obtained in addition to appropriate confidence intervals. In case of missing data, appropriate statistical models will be used investigate differential missing data and analyses where missing data are imputed will be considered as sensitivity analyses. The point estimates with their associated 95 % confidence intervals will be reported. The trial data will be analyzed using SAS 9.4.

Health economic analysis

The aim will be to determine the costs and consequences of low- and high-performing centers. Selected centers will be sent a health economic survey to be completed by local principal investigators. This will micro-cost the unit, looking at staff, capital and consumable costs for colonoscopies and different polypectomies, with patient volume to give an average cost per patient.

We will perform a cost-effectiveness analysis, between high and low performing centers with a short- and long-term analysis. The short-term analysis will compare the number of cancers averted in high versus low performing centers. The long-term analysis will model the costs per quality-adjusted life year of a cohort of individuals, utilizing either a high- or low-performing unit and include the cost of the colonoscopies, CRCs and PCCRCs. Both analyses will allow for an incremental cost-effectiveness ratio to be formulated. A budget impact analysis will be conducted to estimate the financial consequences of improved opadKPI and the earlier detection/prevention of cancers. This will show the cost implications of high vs low performing centers for the NHS. A return on investment tool will be created for commissioners, to help underpin quality-based commissioning.

Process evaluation

After the trial we will undertake a process evaluation using qualitative semi-structured interviews with participants in the intervention arm, to inform interpretation of findings 30 . This will assess how the intervention was implemented and mechanisms of impact purposely sampling for high- and low-performing centers. We will aim to interview a endoscopists who declined recruitment in trial centers and endoscopists in centers which declined to participate in the trial, to understand reasons for non-participation.

Data protection and ethical considerations

NED data collection and use is overseen by the NED committee in accordance with the Royal College of Physicians (RCP) Clinical Standards Department–Information Governance Policy. The relationship between JAG, the data controller, Weblogik, the database supplier and Newcastle University is set out in a contract.

Participation in the trial will be associated with few risks and burdens on endoscopists. Endoscopists may experience stress from performance pressure as their performance is reviewed, however, performance data should currently be routinely monitored locally as part of recommended practice 31 , and all feedback is provided as formative to improve performance and patient care. There are no risks to patients under the care of the participating endoscopists.

Health Research Authority and Health and Care Research Wales approval was given to the study; NHS research ethics committee approval was not required for the trial. Newcastle University Ethics Committee granted approval for all qualitative work. Newcastle-upon-Tyne Hospital Foundation NHS Trust is the sponsor for this study.

Trial management

Trial management committee

The Chief Investigator has overall responsibility for the study and will oversee study management. The Trial Management Group, consisting of the Chief investigator, the national NED clinical lead, representation from the Joint Advisory group for

Gastrointestinal endoscopy and Newcastle University, will be responsible for the day-to-day running of the trial and will meet every 6 months.

Trial period

Recruitment of centers and endoscopists took place from November 2019 to March 2020. Progress with the trial was then interrupted by the COVID-19 pandemic and the halting of endoscopy services. The trial group will monitor UK colonoscopy numbers and once these have increased and plateaued, recruitment of centers will recommence. We estimate delivery of the intervention will start in early 2021, with follow-up for 12 months from completion of recruitment.

Protocol version

The trial is on protocol version 8.0 dated 23 September 2019.

Discussion

We aim to assess whether monthly, automated feedback through NED of endoscopist and endoscopy center-level, case-mix-adjusted performance data will improve colonoscopy performance. A major strength of this RCT is the inclusion of a range of centers from across the UK, including screening and symptomatic services, and the recruitment of colonoscopists from a full range of clinical backgrounds and experience. This maximizes the external generalizibility of the results.

Although ADR is unavailable within NED, our preparatory work indicates that MNP, our opadKPI, captures a breadth of polyp types, is acceptable to endoscopists, and allows procedural adjustment. It will be directly compared with ADR and histological detection KPI at the endoscopy center level.

Due to the nature of the intervention, the trial is not blinded, and may be biased by non-specific effects or “Hawthorne effect” 32 , although we hypothesize this is limited in the control group as interaction is at the point of randomization only. Alternative designs, involving providing all participants with the intervention or with different intensities of intervention, were considered. On balance, it was felt that a two-arm RCT allowing comparison to “usual practice,” i. e. current regular audit and feedback as recommended by JAG, was preferable and maximizes statistical power.

Conclusion

The intervention is evidence-based, theoretically informed, and has been developed with input from endoscopists in routine clinical practice. As far as we are aware, this is the first such feedback intervention to be tested in endoscopy. Our design delivers intervention at both the individual and endoscopy center levels, because endoscopists do not usually work in isolation, and encourages discussing feedback and a community of practice within centers. Nesting out study within NED demonstrates the use of automatically captured data to create entirely automated and hence low cost feedback to healthcare professionals. If the intervention is efficacious and cost-effective, we will showcase the potential of this informatics system to other healthcare sectors and specialties as a highly efficient, cost-effective, and sustainable learning health system that can be implemented at a local and national level. In colonoscopy we hope to demonstrate the effectiveness of this automated system, which collects, analyses, and disseminates real-time clinical data to deliver evidence and theory-informed feedback.