Performance monitoring of EBUS for the staging and diagnosis of lung cancer: auditing the Greater Manchester EBUS service against new national standards.

INTRODUCTION: Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is a pivotal test in lung cancer staging and diagnosis, mandating robust audit and performance monitoring of EBUS services. We present the first regional cancer alliance EBUS performance audit against the new National EBUS specification.
METHODS: Across the five EBUS centres in the Greater Manchester Cancer Alliance, data are recorded at the point of procedure, when pathological results are available and at 6 months postprocedure to review any further pathological sampling (eg, at surgical resection) and the outcome of clinical-radiological follow-up. Outcomes across all five centres were compared with national standards for all lung cancer EBUS procedures from 01 January 2017 to 31 December 2018.
RESULTS: 1899 lung cancer staging or diagnostic EBUS procedures were performed across the five centres during the study period; 1309 staging EBUS procedures and 590 diagnostic EBUS procedures. Major complications were seen in six cases (<1%). All five trusts demonstrated performance above that set national standards in key metrics for both staging and diagnostic EBUS, however the provision of adequate tissue for predictive marker testing was below national standards at one trust. Across Greater Manchester, 72% and 64% of patients had their EBUS procedure performed within 7 days of referral in 2017 and 2018, respectively. Only one out of five trusts met the national targets of >85% of procedures performed within 7 days of referral.
CONCLUSION: The National EBUS service specification is an important framework to drive the quality of EBUS services across the UK. Our data provide assurance of appropriate performance and safety while also highlighting specific areas for attention that can be addressed with the support of the cancer alliance. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Is regular audit of endobronchial ultrasound (EBUS) service especially as outlined in the National EBUS specification beneficial and feasible?

Our audit shows regular performance review of EBUS service assures quality and improved performance of the services.

This is the first audit of EBUS services by a large regional cancer alliance in UK in accordance with the National EBUS specification document showing its utility and feasibility.

Introduction

Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is a bronchoscopic technique that uses real-time ultrasound imaging to guide thoracic lymph node sampling. Over the last 10 years, its increasing use has caused a paradigm shift in lung cancer pathways. The 2019 National Institute for Health and Care Excellence (NICE) lung cancer guidelines recommend EBUS-TBNA for mediastinal staging of lung cancers with any intrathoracic lymph node measuring >10 mm in short axis, and no distant metastases, to provide accurate staging and direct therapeutic planning.1 EBUS-TBNA can also be used to obtain adequate tissue for tumour subtyping and predictive marker testing in selected cases with distant or locally advanced lung cancer.2 3

EBUS services are provided primarily by respiratory physicians and have grown exponentially across UK. The third National Lung Cancer Audit Organisational Audit (covering acute trusts in England and Wales) confirmed that from 94 acute care trusts surveyed, 77% had access to an on-site EBUS service.4 EBUS is a pivotal procedure in the lung cancer pathway that should be performed by appropriately trained operators to ensure high diagnostic accuracy and negative predictive value (NPV). False-negative or inadequate sampling can lead to inaccurate staging risking suboptimal management or delay in the diagnostic pathway potentially necessitating repeat procedures. Therefore, EBUS services require robust monitoring of performance as recommended in the 2019 NICE guidelines.1 These recommendations have been recognised by the Clinical Expert Group for Lung Cancer who have published a National EBUS service specification containing specific quality standards for EBUS outcomes.5

In brief, the National EBUS service specification separates EBUS procedures for lung cancer into two distinct categories; a staging EBUS and a diagnostic EBUS. A staging EBUS is performed to accurately define the presence or absence of nodal metastases in order to map the extent of disease and provide an accurate nodal stage. This requires systematic examination and sampling of the mediastinal and hilar lymph nodes beginning with the nodal stations contralateral to the primary tumour (N3) followed by ipsilateral mediastinal lymph node (N2) stations and then ipsilateral hilar lymph nodes (N1). Diagnostic EBUS, on the other hand, refers to procedures where the aim is to provide adequate tumour tissue for subtyping and predictive marker testing and often involves sampling large frankly malignant lymph nodes. Quality standards are different across the two types of EBUS related to the different underlying aims of the procedure. The most important performance measures of staging EBUS are sensitivity for identifying N2/3 nodal metastases and NPV, both influenced by the false-negative rate. Both sensitivity and NPV have been shown to be dependent on the overall prevalence of N2/3 metastases in the population undergoing EBUS.6–8 It is, therefore, crucial that the prevalence of N2/3 metastases is presented alongside the sensitivity and NPV when considering staging EBUS performance. Diagnostic EBUS performance outcomes are more centred on the adequacy of tumour tissue provided for pathological analysis. The full set of quality standards from the national service specification are provided in tables 1 and 2. This manuscript reports the performance of the five EBUS services across the Greater Manchester (GM) Cancer Alliance measured against these new national standards in the period 2017–2018.

Methods

In Greater Manchester (GM), there are five independent trusts providing EBUS services. One of these trusts (‘trust 1’) uses rapid on site evaluation (ROSE) of cytological samples obtained by EBUS sampling with a cytopathologist present in the EBUS room during procedures. All GM EBUS centres, as a requirement for funding, must commit to complying with the commissioner-led GM EBUS service specification, which was first developed and agreed in 2012 and includes the requirement to submit procedural and outcome data on an annual basis. An annual performance report is submitted to commissioners to monitor access, safety and quality of EBUS across the cancer alliance. This is done via a bespoke standardised database that allows local data entry at the point of procedure as well as the addition of outcome measures when pathological results are available. To provide final outcomes, all procedures undergo a 6-month postprocedure review with further outcomes added to the database. For sensitivity and NPV calculations, the identification of patients with N2/3 metastases missed by systematic staging EBUS is pivotal. This requires a thorough review of any subsequent pathological nodal sampling (eg, mediastinoscopy or intraoperative lymph node sampling) and a minimum of 6 months clinical and radiological follow-up. Individual procedure data and outcomes are entered by the local EBUS teams and completed databases are submitted to a central team for analysis in the Cancer Alliance. Data are submitted to the central team 6 months after the last EBUS procedure within the audit period and then requires a period of time to address data queries, data cleansing and analysis, for example, 2018 EBUS data are submitted at the mid-way point of 2019 resulting in final publication of the performance report in late 2019. In this manuscript, the 2017 and 2018 EBUS data for each of the five centres were compared with the quality standards set out in the national service specification. Given that EBUS data submission has occurred since 2012 in GM, we have also presented performance over time for this 6-year period where possible noting that initially just three centres contributed to the data submission from 2012 to 2016 and diagnostic EBUS outcome data were only submitted from 2016.

Patient and public involvement

There was no direct patient and public involvement in the design of this study as it is a service evaluation. The results of this study are presented at the Cancer Alliance Group that has patient and public representatives. The results help us to improve the service specifications in accordance with the set National EBUS performance standards.

Ethics statement

This is an audit for quality improvement and monitoring. All patients attending EBUS procedures sign a consent form, which includes consent for using the data for audit purposes by the department. Given this, ethical approval for this work was not needed, confirmed by discussion with the local ethics committee.

Results

Between 01 January 2017 and 31 December 2018, 3051 EBUS procedures were performed across GM. This included 1899 lung cancer-specific EBUS procedures; 1309 staging EBUS procedures and 590 diagnostic EBUS procedures. The number of EBUS operators at each centre for this time period was centre 1: four operators, centre 2: four operators in 2017 and five operators in 2018, centre 3: one sole operator in 2017 and two operators in 2018, centre 4: seven operators and centre 5: three operators. The median time from referral to procedure across GM was 6 days in 2017 and 2018. Overall, GM did not meet the national standard of >85% of EBUS procedures performed within 7 days of referral; 72% and 64% of patients had their EBUS procedure performed within 7 days of referral in 2017 and 2018, respectively. Only one trust (trust 3) consistently met the target of >85% within 7 days of referral (table 3). The median time for pathology turnaround from EBUS procedure to morphology and immunohistochemistry (IHC) was 3 days and 2 days in 2017 and 2018. Of 83% and 81% of pathology results were available within 3 days of EBUS in 2017 and 2018, respectively. Two trusts (trusts 2 and 4) are consistently achieving the national target of a pathology turnaround time of 3 days or less in >85% of EBUS procedures (table 3). There were six major complications from 3051 procedures in the 2-year period (0.2%). These major complications were oversedation requiring reversal agents (n=3), pneumothorax requiring intervention (n=1), cardiac arrhythmia requiring intervention (n=1) and an unplanned hospital admission (n=1). All five trusts meet the national standard of a major complication rate of ≤3%.

Table 3

Waiting times, pathology turnaround times and safety of EBUS across the five GM EBUS centres and comparison to national standards

	Trust 1		Trust 2		Trust 3		Trust 4		Trust 5
	2017n=268	2018n=331	2017n=92	2018n=96	2017n=238	2018n=318	2017n=243	2018n=223	2017n=41	2018n=49
Median time referral to EBUS (days)	6	9	8	8	6	5	5	6	1	5
EBUS performed within 7 days	65%	37%	44%	48%	86%	86%	84%	76%	81%	87%
Median time EBUS to pathology (days)	1	0	5	5	9	3	3	3	5	5
Pathology result within 3 days	94%	94%	68%	67%	67%	68%	87%	86%	52%	63%
Major complication rate	0%	1%	0%	0%	0%	3%	0%	0%	0%	0%

EBUS, endobronchial ultrasound; GM, Greater Manchester.

Staging EBUS

There was variability in the rate of staging EBUS procedures where any of the lymph nodes sampled were deemed pathologically inadequate (range 1%–18%, table 4). However, sensitivity and negative predictive were almost universally above those set out in the national quality standards, stratified according to the prevalence of N2/3 disease. The only notable exception was at trust 5 where sensitivity was 50% in 2017.

Table 4

Performance of staging EBUS across GM and comparison to national quality standards

	Trust 1four operators		Trust 2four operators 2017five operators 2018		Trust 3one operator 2017two operators 2018		Trust 4seven operators		Trust 5two operators
	2017	2018	2017	2018	2017	2018	2017	2018	2017	2018
Total number of staging EBUS	170	209	66	72	120	291	189	142	22	28
Number of inadequate staging procedures	1%	2%	18%	12%	7%	16%	6%	2%	17%	15%
Sensitivity(target)	92%(>80%)	90%(>80%)	82%(>85%)	93%(>85%)	96%(>85%)	93%(>88%)	85%(>85%)	86%(>80%)	50%(>75%)	82%(>80%)
Negative predictive value(target)	97%(>90%)	96%(>90%)	89%(>85%)	95%(>85%)	94%(>85%)	90%(>80%)	95%(>90%)	93%(>90%)	89%(>90%)	88%(90%)
Prevalence of N2/N3 disease	29%	30%	41%	43%	58%	61%	41%	35%	18%	39%
Mean number of lymph nodes sampled per procedure	2.8	2.9	2.1	2.0	2.3	2.4	2.9	2.8	1.8	2.0

EBUS, endobronchial ultrasound.

Diagnostic EBUS

The pathological confirmation rate was good across the audit period with only trust 2 performing below the national target of >90% in 2017, which has improved to above the national standard in 2018 (table 5). The nonsmall cell lung cancer (NSCLC) not otherwise specified (NOS) rate was above that expected within the national standards (<10%) at trusts 2, 3 and 5 ranging from 15% to 35%. Furthermore, the provision of adequate tissue for epidermal growth factor receptor (EGFR) mutation testing and anaplastic lymphoma kinase (ALK) rearrangement mutation testing at trust 2 was well below the national target of >90% across both audit periods (range 50%–64%). Trust 5 also did not meet the national standard for EGFR and ALK testing at several points in the audit although by a very small margin (range 80%–89%).

Table 5

Performance of EBUS in the diagnosis of lung cancer across GM and comparison to national quality standards

	Trust 1four operators		Trust 2four operators 2017five operators 2018		Trust 3one operator 2017two operators 2018		Trust 4seven operators		Trust 5two operators
	2017	2018	2017	2018	2017	2018	2017	2018	2017	2018
Number of diagnostic EBUS	98	122	26	24	118	27	54	81	19	21
Pathological diagnosis	91%	96%	80%	91%	98%	93%	93%	91%	94%	95%
NSCLC-NOS rate	3%(4/154)	4%(7/193)	25%(11/44)	21%(8/39)	2%(4/214)	35%(68/197)	7%(8/113)	8%(8/98)	17%5/29	15%4/27
Adequate tissue for EGFR	91%(89/98)	96%(117/122)	55%(12/22)	57%(8/14)	99%(93/94)	90%(63/70)	90%(55/61)	90%35/39)	87%13/15	94%17/18
Adequate tissue for ALK	98%(96/98)	97%(89/92)	64%(16/22)	50%(6/12)	100%(30/30)	90%(62/69)	90%(51/57)	93%37/40	80%12/15	89%16/18

EBUS, endobronchial ultrasound; EGFR, epidermal growth factor receptor; NOS, not otherwise specified; NSCLC, nonsmall cell lung cancer.

EBUS performance over time

Performance in staging EBUS across trusts 1,2 and 4 is presented in figure 1. In all three centres, the prevalence of N2/3 in those undergoing EBUS staging has reduced over time and stabilised around 30%–40%. All trusts have shown improved performance in staging EBUS over time, particularly trusts 1 and 2. At trust 1, the NPV has increased from 68% in 2014 to consistently over 95% and at trust 2; sensitivity has improved from 59% in 2012 to 93% in 2018 alongside an improvement in NPV from 72% to 95%. Trust 4 has consistently shown high performance from 2012 to 2018. In diagnostic EBUS, trusts 1 and 4 have shown very high performance consistently from 2016 to 2018 and trust 2 has made significant improvements in pathological confirmation rates, from 77% in 2016 to 91% in 2018, but concern remains over the level of adequate tissue for EGFR and ALK testing as well as the NSCLC-NOS rate (figure 2).

Figure 1

Staging EBUS performance 2012–2018 for (A) trust 1, (B) trust 2 and (C) trust 4. EBUS, endobronchial ultrasound; NPV, negative predictive value.

Figure 2

Diagnostic EBUS performance 2016–2018 for (A) trust 1, (B) trust 2 and (C) trust 4. EBUS, endobronchial ultrasound; EGFR, epidermal growth factor receptor; ALK, anaplasticlymphoma kinase; NSCLC, nonsmallcell lung cancer; NOS, not otherwisespecified.

Discussion

Key findings

This audit of performance across five EBUS services in GM has provided assurance of high-quality EBUS provision in a number of key metrics. Performance in staging EBUS is almost universally above the standards set out in the national service specification across the five centres and assures the commissioners that appropriate nodal staging to inform prognosis and treatment decisions is taking place. The only notable exception was at trust 5 where sensitivity was 50% in 2017. It should, however, be noted that this is a low volume centre and the prevalence of N2 was very low in this time period (only four patients with N2/3 disease) and should be interpreted with caution, especially as performance was above national targets the following year when the prevalence of N2/3 was increased. In diagnostic EBUS, there are very high rates of pathological confirmation of advanced lung cancer across all five providers and high levels of predictive marker completion in four out of five EBUS units. The audit has, however, identified several areas for improvement. First, adequate tissue for predictive marker testing is well below the national standard at trust 2. The results of this audit have triggered the formation of a dedicated EBUS working group across the respiratory and pathology teams at trust 2 to focus on the acquisition of appropriate volume of tumour tissue during the procedure as well as improving the handling processes in the pathology department. They will review results in real time to ensure improvements are made and this will continue to be reviewed in future GM EBUS performance reports. These results have also triggered consideration of the high number of operators given the lower number of total procedures at this trust and the potential advantages of increasing experience and expertise across a smaller number of operators. It was also noted that rates of NSCLC-NOS are high across three trusts and this has been fed back to pathology departments to review internally and consider their IHC processes as subtyping of NSCLC helps inform the choice of chemotherapy in advanced disease with pemetrexed based chemotherapy the standard of care in nonsquamous NSCLC.1 This will also be reviewed in future GM EBUS performance reports. The use of ROSE in EBUS services is a topic of debate internationally. In GM, there is one centre that uses ROSE, which delivers consistently high performance (centre 1, tables 4 and 5). Positive feedback on the value ROSE from this centre and its referral teams include same day initial results and the early identification of possible small cell to facilitate rapid onward referral. However, equally high performance has been demonstrated in other non-ROSE centres and final results that include IHC findings are always required to finalise onward management and these factors could question the added value of the resources needed to deliver ROSE. Currently in GM, there are no plans to expand the use of ROSE. The most consistent area of substandard performance was in timely access to EBUS procedures with only one out of five centres achieving the national standard of 85% of procedures completed within 7 days of referral. This has prompted consideration of a ‘single queue’ EBUS service for GM in which patients are offered the next available appointment across the cancer alliance with an agreed EBUS provider rather than waiting for their next local service appointment. Further work in developing this proposal is ongoing and will be presented to the GM cancer system and commissioners for consideration. Finally, this work highlights the benefit of performance monitoring and engaging with commissioners to improve outcomes over time. This is now very clear to see when data are reviewed over the time period of this regional EBUS service review. For staging EBUS performed in GM from 2012 to 2018, the prevalence of N2/3 disease in those undergoing EBUS has decreased and then stabilised at approximately 30%–40% at most centres. This suggests increased staging of patients with a low prevalence of N2/3, such as those with a normal mediastinum radiologically (eg, those with N1 disease or a central tumour with a normal mediastinum). This not only represents increasing skill level of EBUS operators and more challenging staging procedures completed but also represents the appropriate population having EBUS as part of the optimal pathway. Despite the increasing skill level required, all centres have shown increased or maintenance of high performance over the 6 years. This once again highlights the pivotal importance of performance monitoring as a vehicle for improving outcomes.

Strengths of the study

This level of performance monitoring of EBUS in lung cancer at a regional level is unique across the UK and has been highlighted as a national exemplar by the UK Lung Cancer Coalition in its publication ‘Pathways Matter’.9 This study provides strong evidence to support monitoring of cancer investigation services such as EBUS-TBNA at a cancer alliance level and be an integral component of a commissioner-led regional service specification. The results of all annual GM EBUS performance reports are discussed at the cancer alliance Lung Pathway Board and the GM Directors of Commissioning meetings. Submission of outcome data is a contractual requirement for all EBUS services to secure ongoing commissioning as well as the development of an appropriate action plan to address any areas where performance falls below the expected standard. This formalised process supports local teams to engage with their trust senior management team to address areas of concern with additional resource to ensure improvement in the required areas. Early engagement of all EBUS centres in the initial development of the data fields and service specification back in 2012, and which has continued ever since, has been an important part of the success of this project. Furthermore, this engagement has always been across the breath of the team responsible for the test performance, not chest physicians alone but the pathology team and bronchoscopy nursing staff who were always represented on the regional EBUS working group.

This is the first regional EBUS service to publish its performance measured against the new National service specification quality standards. This national document is welcomed by the GM cancer alliance as a vehicle to replicate comprehensive performance monitoring at scale across the UK.

Limitations of the study

The data quality is reliant on the local EBUS team and, therefore, opens to entry error and investigator bias. This must, therefore, be acknowledged when interpreting the data. Unfortunately, this is the only methodology available for this work as standardised reporting software, which combines procedural data, pathological results and verification information from intraoperative lymph sampling or clinical–radiological follow-up across multiple trusts does not exist. Furthermore, there is an inherent delay in data submission due to the requirement for the 6-month follow-up data used to confirm the final outcome of EBUS and inform performance measures. A further wait for data queries, data cleansing and data analysis are then required prior to publication, meaning the results we are reviewing and actioning are not the most up to date data for each service. Not all performance outcomes require the 6-month review, for example, pathway metrics and diagnostic EBUS outcomes, and these quality standards could be measured in a more timely way. This is something under discussion and review within our cancer alliance. It is also apparent that the bespoke GM database does not capture all the required data to report against every standard set out in the National service specification. We will adapt our data collection to include the adequacy of tissue for ROS-1 rearrangement and Programmed Death Ligand 1 testing, the number of diagnostic EBUS procedures that require a further sampling procedure due to inadequate tissue provision and the number of procedures (including a number of staging procedures) performed by individual operators at each centre. This is also the opportunity to consider a more robust data collection method that is not a bespoke database but an embedded NHS web-based system, which could support the single queue referral process and also communication of results as well as data capture for performance monitoring. Investment in dedicated data managers to support this regional service would enhance data completion and improve the speed of analysis and feedback and reduce the impact on clinicians.

Conclusion

Efficient access to high-quality EBUS services is paramount to facilitate the implementation of the National Lung Cancer Optimal Pathway and achieving new national cancer targets such as the 28 day faster diagnosis standard.10 The National EBUS service specification is an important framework to drive the quality of EBUS services across the UK. Furthermore, we strongly believe a local commissioner-led annual quality assurance and audit across a cancer alliance further drives service improvements and helps identify specific areas for attention that can be addressed at a regional and local level with the support of the cancer alliance.

Table 1

Quality standards in the National EBUS service specification5

Quality performance indicator	Threshold
Procedure carried out within 7 working days of receipt of referral	85%
Pathological results received within 3 working days of receipt of samples**This includes morphology and four panel immunohistochemistry	85%
Total pathway time—10 working days (from referral to receipt of pathology results*)*This includes morphology and four panel immunohistochemistry	85%
Safety—major/minor complications	<3% major
Proportion of procedures where any lymph node station was inadequate	<10%
SensitivityDenominator=total number of patients with N2/3 metastases	Based on prevalence of N2/N3 disease
Negative predictive valueDenominator=total number of patients with a negative staging EBUS for N2/3	Based on prevalence of N2/N3 disease
Prevalence of N2/3 nodal metastases in population	%
Pathological confirmation rate in advanced disease	>90%
Adequate tissue for successful EGFR testing	>90%
Adequate tissue for successful ALK testing	>90%
Adequate tissue for successful ROS-1 testing	>90%
Adequate tissue for successful PD-L1 testing	>90%
NSCLC-NOS rate	<10%
Proportion of cases in which a repeat sampling procedure is needed due to insufficient tissue*(* Does not include patients in which core tissue is needed for clinical trail.)	<10%

ALK, anaplastic lymphoma kinase; EBUS, endobronchial ultrasound; EGFR, epidermal growth factor receptor; NOS, not otherwise specified; NSCLC, nonsmall cell lung cancer; PD-L1, programmed death ligand 1; ROS-1, ros UR2 sarcoma virus oncogene homolog 1.

Table 2

Recommended minimum standards for staging EBUS according to the prevalence of N2/3 nodal metastases in the population undergoing EBUS5 7

N2/3 prevalence	Sensitivity		Negative predictive value
	ACCP meta-analysis	Minimum standard	ACCP meta-analysis	Minimum standard
>80%	96%	>90%	83%	>80%
60%–80%	91%	>88%	83%	>80%
40%–60%	87%	>85%	89%	>85%
20%–40%	87%	>80%	95%	>90%
<20%	78%	>75%	96%	>92%

ACCP, American College of Chest Physicians; EBUS, endobronchial ultrasound.