Multiple breath washout in bronchiectasis clinical trials: is it feasible?

BACKGROUND: Evaluation of multiple breath washout (MBW) set-up including staff training, certification and central "over-reading" for data quality control is essential to determine the feasibility of MBW in future bronchiectasis studies. AIMS: To assess the outcomes of a MBW training, certification and central over-reading programme.
METHODS: MBW training and certification was conducted in European sites collecting lung clearance index (LCI) data in the BronchUK Clinimetrics and/or i-BEST-1 studies. The blended training programme included the use of an eLearning tool and a 1-day face-to-face session. Sites submitted MBW data to trained central over-readers who determined validity and quality.
RESULTS: Thirteen training days were delivered to 56 participants from 22 sites. Of 22 sites, 18 (82%) were MBW naïve. Participant knowledge and confidence increased significantly (p<0.001). By the end of the study recruitment, 15 of 22 sites (68%) had completed certification with a mean (range) time since training of 6.2 (3-14) months. In the BronchUK Clinimetrics study, 468 of 589 (79%) tests met the quality criteria following central over-reading, compared with 137 of 236 (58%) tests in the i-BEST-1 study.
CONCLUSIONS: LCI is feasible in a bronchiectasis multicentre clinical trial setting; however, consideration of site experience in terms of training as well as assessment of skill drift and the need for re-training may be important to reduce time to certification and optimise data quality. Longer times to certification, a higher percentage of naïve sites and patients with worse lung function may have contributed to the lower success rate in the i-BEST-1 study.

Introduction

There is a need for more reliable and responsive measures of lung function in bronchiectasis. Lung clearance index (LCI) derived from multiple breath washout (MBW) is a robust measure of ventilation distribution and studies in bronchiectasis have demonstrated LCI to be a reliable, valid and sensitive measure of lung disease [1, 2]. With emerging evidence to support its use and the need for further data on long-term reliability and responsiveness, LCI has been adopted as an exploratory end-point in two large multicentre studies in bronchiectasis (BronchUK Clinimetrics and i-BEST-1). The BronchUK Clinimetrics study is a longitudinal observational study to explore the clinimetric properties of a range of outcome measures to support recommendations for bronchiectasis clinical studies (ClinicalTrials.gov Identifier: NCT02468271). The Innovative Medicines Initiative (IMI) sponsored i-BEST-1 study is a phase II dose-finding study, to assess the efficacy, safety and tolerability of tobramycin inhalation powder in patients with noncystic fibrosis (CF) bronchiectasis and pulmonary Pseudomonas aeruginosa infection (ClinicalTrials.gov Identifier: NCT02712983) [3]. MBW testing to measure LCI is not a core clinical research skill and effective training is essential for data quality control [4, 5]. Acquisition of good quality MBW data can be influenced by operator training, competence and experience in testing, as well as patient factors (e.g. familiarity with the test, ability to cooperate and tolerate the test procedure [6, 7]). In addition to staff training and certification, central “over-reading” of the data collected during the study is recognised as an important step for quality control, especially in the multicentre setting [5, 8]. Central over-reading ensures that standardised, pre-defined criteria are used in the evaluation and analysis of the data [7]. In other studies, a range of success rates have been reported with test failure attributed to both training/site and patient-related reasons and it is important to assess their impact in the bronchiectasis multicentre research setting. Determination of the feasibility of MBW in bronchiectasis studies is required.

Objectives

The aims of this study were: 1) To evaluate the training and certification processes by assessing the change in participant knowledge and confidence post-training and participant feedback; and certification rates, time elapsed between training and certification and reasons for failure to certify. 2) To summarise the outcomes of central over-reading by exploring the proportion of tests excluded after central over-reading and most common reasons; and success rates per site, comparing naïve and non-naïve sites and change in success rate over time. 3) To summarise the time and staff capacity required to complete MBW training, certification and over-reading.

Methods

Clinical trial sites

Sites completed a MBW feasibility checklist (supplementary appendix 1). Twenty-eight European sites (UK, Spain, Germany, France, Italy, Belgium, Ireland) participated in the LCI substudy of the i-BEST-1 study. Seven of these sites also participated in the BronchUK Clinimetrics study. Both studies used the Exhalyzer® D Multiple Breath Nitrogen Washout system and the European CF Society (ECFS) standard operating procedure. Studies were prospectively registered and data collection was approved by the Research Ethics Committee MR/L011263/1 (Clinimetrics) and the country-specific independent Research Ethics Committee (i-BEST-1 study).

Training and certification

A series of training programmes were delivered by 1–2 clinical researcher(s) from the Belfast centre certified and experienced in MBW testing using the Exhalyzer® D system (supplementary appendix 2). For certification, sites were required to collect and submit 10 MBW testing sessions (5 per operator), including 4 from healthy people and 6 from people with respiratory disease (ECFS MBW Central Training and Over-Reading Centre criteria, permission granted) (supplementary appendix 3).

Knowledge and confidence assessment and programme evaluation

Participants completed a knowledge and confidence questionnaire before receiving eLearning tool access and the preparation material, at the start and at the end of the training day (supplementary appendix 4). Additionally, participants completed an evaluation on programme content, format and delivery (North American MBW Training Centre, permission granted).

Quality control and over-reading of study data

During the studies, the de-identified MBW data (each subject assigned a unique study ID) were over-read [7], cleaned and double over-read by trained personnel (supplementary appendix 5).

Time estimate

The time and staff capacity (hours and days) for training, certification and over-reading activity was estimated based on the experiences of the Belfast centre in these studies.

Statistical analysis

The characteristics of the training programme participants were summarised. Data distribution of all variables of interest was tested and the appropriate parametric and nonparametric statistics were applied. Differences between the i-BEST-1 and BronchUK Clinimetrics study were compared using nonparametric statistics (p<0.05 = statistically significant). To assess site-related impacts on data quality control, a Wilcoxon signed-rank test was used to compare naïve and non-naïve sites and between the first 6 months versus subsequent 6 months of testing. To assess patient-related impacts on data quality control, Mann–Whitney U-tests were used to compare differences with subject age, lung function and pulmonary exacerbation (where data were available i.e. Clinimetrics study). Combined success rate data from both studies (i-BEST-1 and BronchUK Clinimetrics) were compared using the Mann–Whitney U-test.

Results

Training programme

Thirteen training days were delivered to 56 participants from 22 of the 28 sites (11 in the UK, 11 in other countries in Europe) over the course of 25 months (April 2016 to May 2018). The remaining 6 of the 28 sites had previously completed an ECFS MBW training and certification process for other studies in CF patient populations. Of the 22 sites that completed training, 18 (82%) were MBW naïve (no previous knowledge or experience in MBW testing).

Participants in the training programmes were from a variety of professional backgrounds and included nurses (21 of 56; 38%), physicians (9 of 56; 16%), clinical physiologists (9 of 56; 16%), study coordinators (9 of 56; 16%) and physiotherapists (8 of 56; 14%).

Comparing the studies, 3 of 7 (43%) of the sites in the BronchUK Clinimetrics study were MBW naïve, compared to 73% (8 of 11) in the i-BEST-1 study. All 9 physicians were from sites in the i-BEST-1 study, otherwise, there was no difference in the proportion of all other professional backgrounds between the two studies.

Knowledge and confidence assessment

Knowledge and confidence questionnaire scores are summarised in supplementary appendix 6. At baseline, participants had some knowledge of MBW testing (mean score 6.0/10); however, confidence was low. Knowledge and confidence increased significantly over the self-directed learning period and further between start and end of the programme. The largest increase was observed after the 1-day face-to-face training programme. Forty-six out of the 56 participants (82%) planned to continue using the eLearning tool after the training day and throughout both studies (April 2016 to June 2019) the publicly available eLearning tool had 8602 page views. Use of the eLearning tool was in a bite-size fashion with users viewing 3 pages per session and an average session duration of 3 min (Google Analytics).

Comparing the studies, there was no difference in the change in participant knowledge or confidence (p>0.05).

Programme evaluation

Fifty-two (93%) of the 56 participants completed the programme evaluation. Full programme evaluation results and participant feedback are detailed in appendix 7.

Certification

By the end of the recruitment phase of both studies, 15 of 22 (68%) of the sites had completed certification with a mean±sd (range) time since training of 6.2±2.9 (3–14) months. Time to certification was significantly longer in i-BEST-1 (7.4 months) compared to Clinimetrics (4.3 months) (p=0.02).

Six (40%) sites completed their certification on the first submission (1 in Clinimetrics, 4 in i-BEST-1, 1 in both), eight (53%) completed on the second submission (4 in i-BEST-1, 4 in both) and 1 (7%) required three submissions (1 in i-BEST-1).

The remaining sites (7 of 22; 32%) did not complete certification. Three withdrew from the certification process (2 sites with inadequate staff time to complete the certification, 1 later found clinical space was unsuitable for LCI equipment). The remaining 4 sites (1 submitted but failed, 3 did not submit any data) did not adequately complete certification before the end of the recruitment phase, reporting the barriers as listed in supplementary appendix 8.

Data quality control and central over-reading

Overall, of the 22 sites trained, only 12 actually recruited participants and collected MBW data in the studies.

In the Clinimetrics study, trial participants attempted MBW at 8 visits over 12 months, 589 MBW testing sessions (n=1571 tests) from 142 study participants across seven sites were submitted for over-reading. After completion of over-reading (supplementary appendix 5), 468 of 589 (79%) testing sessions met the quality criteria, whereas 121 of 589 (21%) testing sessions did not and were excluded (figure 1). Site success rates across sites ranged from 61% to 89%. Success rates in MBW-naïve sites was 72% compared to 85% in non-naïve sites. A decrease in success rate after 6 months was observed in naïve sites compared with non-naïve sites, indicating that skill drift may have occurred (table 1). Subjects attempting MBW had a mean±sd age of 65.9±11.1 years, forced expiratory volume in 1 s (FEV1) of 71.5±19.7 % predicted and 2.5±2.1 pulmonary exacerbations in the previous year. In those patients with a valid MBW at baseline (n=131), mean±sd LCI was 12.4±3.0. Considering patient-related reasons for test exclusion, there was no difference in age (p=0.10) or disease severity (p=0.90) as measured by FEV1 (at baseline) in those subjects with successful versus those with failed test sessions in this cohort.

FIGURE 1

Central over-reading outcome from BronchUK Clinimetrics study (testing sessions from 142 subjects over 8 visits). MBW: multiple breath washout. #: re-run of tests completed but did not resolve the misalignment or re-run not possible due to A-files not being saved.

TABLE 1

Clinimetrics (Clin) multiple breath washout data quality control according to site

	Naïve sites			Non-naïve sites				Overall
Site	Clin site 1	Clin site 2	Clin site 3	Clin site 4	Clin site 5	Clin site 6	Clin site 7
Total number of testing sessions attempted	116	59	58	88	91	158	19
Site success rate	81/116 (70%)	36/59 (61%)	50/58 (86%)	65/88 (72%)	84/91 (92%)	140/158 (89%)	12/19 (63%)	468/589
Combined success rate naïve versus non-naïve	167/233 (72%)			301/356 (85%)				79%
Site success rate first 6 months versus remainder	81%:65%	81%:53%	94%: 83%	74%:74%	72%:100%	94%:88%	64%:60%

In the i-BEST-1 trial, subjects attempted MBW at 8 visits over 6 months. Overall, 236 MBW testing sessions (n=675 tests) from 40 study participants across 11 sites were submitted for over-reading. After completion of the over-reading process (supplementary appendix 5), 137 of 236 (58%) testing sessions met the quality criteria, whereas 99 of 236 (42%) testing sessions did not and were excluded (figure 2). Success rates per site ranged from 0 to 81%. Three of the 11 sites were unable to collect any valid data from patients (i.e. all tests excluded). On exclusion of the data from these sites, the proportion of testing sessions excluded overall was reduced to 36%. In contrast to what was observed in the Clinimetrics study, in the i-BEST-1study success rates in MBW-naïve sites was 59% compared to 56% in non-naïve sites. In addition in the i-BEST-1 study there was an increased success rate after 6 months in some naïve sites, indicating that there was a learning effect and that increased experience with testing may have improved test quality (table 2). Subjects attempting MBW had a mean±sd age of 64.1±13.0 years, FEV1 54.8±19.0 % predicted and 2.6±1.6 pulmonary exacerbations in the previous year. In those patients with a valid MBW at baseline (n=19), mean±sd LCI was 13.9±3.5.

FIGURE 2

Central over-reading outcome from i-BEST-1 trial (testing sessions from 40 subjects over 8 visits). MBW: multiple breath washout. #: re-run of tests completed but did not resolve the misalignment or re-run not possible due to A-files not being saved.

TABLE 2

i-BEST-1 multiple breath washout (MBW) data quality control according to site

	Naïve sites							Non-naïve sites				Overall
Site	i-BEST site 1	i-BEST site 2	i-BEST site 3	i-BEST site 4	i-BEST site 5	i-BEST site 6	i-BEST site 7	i-BEST site 8	i-BEST site 9	i-BEST site 10	i-BEST site 11
Number of testing sessions attempted	20	48	4	65	12	20	10	4	32	8	13
Overall success rate	6/20 (30%)	35/48 (73%)	0/4 (0%)	34/65 (52%)	9/12 (75%)	15/20 (75%)	6/10 (60%)	0/4 (0%)	26/32 (81%)	6/8 (75%)	0/13 (0%)	137/236
Combined success rate naïve versus non-naïve	105/179 (59%)							32/57 (56%)				58%
Success rate first 6 months versus remainder	46%:0%	68%:91%	0%:0%	48%:55%	75%:n/a	93%:100%	60%:n/a	0%:0%	84%:72%	86%:0%	0%:0%

n/a: No MBW data submitted during this timeframe.

Figure 3 displays combined success rate data from both studies considering naïve and non-naïve status. Success rates for sites participating in both studies were averaged. Success rates were higher in non-naïve sites (81%) compared to naïve sites (66%).

FIGURE 3

Combined success rate. Clin: Clinimetrics.

The time and staff capacity required completing MBW training, certification and over-reading is summarised in supplementary appendix 5.

In summary, the key results from this study are displayed in table 3.

TABLE 3

Key results from multiple breath washout (MBW) training, certification and data quality control in bronchiectasis research settings

Knowledge and confidence with MBW testing was increased with the training programme

68% of sites completed certification within 6 months

Most common barriers to completing MBW certification in bronchiectasis sites were the lack of time and available volunteers to practice testing

Success rates were 79% (Clinimetrics study) and 58% (i-BEST-1 study)

In the Clinimetrics study, there was no difference in age (p=0.10) or disease severity (p=0.90) as measured by forced expiratory volume in 1 s (at baseline) in those subjects with successful versus those with failed test sessions

In the i-BEST-1 study, longer times to certification, a higher percentage of naïve sites and patients with worse lung function may have contributed to the lower success rate

Combination of success rate data from both studies showed that success rates were higher in non-naïve sites (81%) compared to naïve sites (66%)

Discussion

This study demonstrated that MBW testing to measure LCI, a newly emerging, repeatable and sensitive outcome measure in bronchiectasis is feasible in a bronchiectasis multicentre clinical trial setting. However, key challenges remain regarding time to site certification and in obtaining the optimal proportion of valid data in this patient population. Longer times to certification and patients with worse lung function may have contributed to the lower success rate in the i-BEST-1 study. Success rate data from both studies showed higher success rates in non-naïve sites compared to naïve sites. To optimise data quality this study highlights that the steps of training, certification and quality control are essential components to consider at the trial design and site feasibility stage. Consideration of site experience in terms of training needs as well as assessment of skill drift and the need for re-training may be important to reduce time to certification and optimise data quality. MBW success rates can be linked to both training-related and patient-related reasons and we have attempted to assess their impact in the bronchiectasis multicentre research setting.

Effect of training and certification

Regarding training, our knowledge and confidence evaluation highlighted the importance of the face-to-face element of training. MBW knowledge but particularly confidence was significantly increased after this component of the programme. Hands-on practice with calibration and testing with the equipment has been emphasised as an essential element in any MBW training, including other reports of 1- and 2-day programmes [9, 10]. While 10% of participants suggested that covering all content in 1 day was a challenge and more time for hands-on practice may be better, the high level of participation satisfaction with programme delivery and content generally supports this model of training. Conversely, a second day may have increased the opportunity to be trained and reduced the number of sites failing to be certified; however, there may be challenges in getting staff released from normal duties for a second day of training. The use of our eLearning tool was an integral component of the teaching and in addition was designed to provide programme participants with refresher content on return to their home site. Use of such a tool is relevant in the multicentre research setting to facilitate open and continuous access to training materials.

In the Clinimetrics study 3 of 7 (43%) were naïve to MBW testing, in contrast to 73% (8 of 11) in the i-BEST-1 study. Our data found no difference in the change in participant knowledge or confidence with training (p>0.05) between the studies. However, our data showing higher success rates from the combined naïve versus non-naïve sites are in agreement with our previous study demonstrating that site MBW experience relates to data quality [6]. Furthermore, data from the i-BEST-1 study showed evidence of a learning effect, with some sites’ success rates improving after 6 months. These results may be helpful in informing site selection criteria for future studies including LCI (i.e. previous experience >6 months). However, this was in contrast to the skill drift seen over 6 months in the Clinimetrics study. Additionally, 6 sites were active in both studies, some at the same time, and therefore the pattern is unclear. The best way to measure site experience is unclear and quantification of operator testing hours may be a better indicator of MBW experience.

Our finding that sites required an average of 6 months to complete both training and certification is an important consideration for sites and sponsors planning future MBW bronchiectasis studies. A longer time to certification may contribute to a higher test failure rate due to skill drift, as shown in the i-BEST-1 study. The most common barriers to certification (lack of time, lack of available volunteers for practice testing) were in agreement with challenges highlighted by Central Over-Reading Centres in CF [11]. Consideration of language translational needs is also important. More detailed criteria on time required for certification at the feasibility assessment stage may be helpful. Furthermore, reducing theory content and organising at least 1 practice test per operator during the face-to-face training day may facilitate the process. Other studies have included supervised MBW assessments at site initiation [12]. Post-certification, in addition to the site support provided (supplementary appendix 3) these study results emphasise the need for more protocolised assessment of skill drift to identify sites with underperformance earlier, optimise shared learning and target sites for more formalised booster re-training. Such an approach may have prevented skill drift and future studies should consider such strategies. All potential solutions and activity require planning and resourcing by the centre delivering the training. Key differences in the bronchiectasis research setting compared to the CF setting (staff are not usually bronchiectasis-specific or based in a dedicated team or centre) should be considered.

Effect of study design

Differences in success rates may also be influenced by study design. In the Clinimetrics study, LCI was a requirement at baseline and repeat visits were facilitated when data were invalid. In the i-BEST-1 study, recruits attended 8 visits with a greater number of assessments over 6 months. More complex study design may have placed time constraints on the LCI testing period and made additional visits for repeat LCI unfeasible. Overall, in the i-BEST-1 study, while there were fewer patients and fewer testing opportunities, there were more frequent study visits (every 2–4 weeks) compared to the Clinimetrics study (every 3 months). This more frequent practice may explain the learning effect seen.

Effect of patient-related reasons for exclusion

Considering patient-related reasons for test exclusion, overall patients in i-BEST-1 had a worse FEV1 % predicted compared with the Clinimetrics study (although no association was found between FEV1 and LCI success within studies). Worse lung function and lower tolerability can mean longer testing time and duration of the MBW test is a key determinant of feasibility [4] and this may have been a contributing factor in the lower success rates seen in i-BEST-1. Exploration of parameters of MBW that involve less time and patient burden (e.g. shortened LCI) may be warranted [13, 14].

The largest proportion of tests excluded in both studies was due to a leak during testing. Maintaining a tight seal with the mouthpiece and nose clip in the same position can be challenging. This is particularly problematic if the patient is symptomatic or during a pulmonary exacerbation. Furthermore the worse the lung function, the longer the test, increasing the likelihood of leak. Leaks have also been the main reason for test exclusion in CF studies reporting on MBW quality control, highlighting that it remains an issue regardless of patient population [7, 15]. During training, operator recognition of leaks and steps to resolve them are a focus and continued emphasis on this aspect in future programmes is important to minimise excluded tests.

Bronchiectasis versus CF success rates

Success rates in the Clinimetrics study were comparable to that reported in recent CF multicentre studies and in another single-centre study of LCI in bronchiectasis (success rate of 73–98%) [11, 12, 16–21]. Lower success rates reported in the i-BEST-1 study (58%) demonstrate the challenges of MBW testing in this setting, despite the approach of central training, implementation of standard operating procedures and harmonised MBW protocols and central MBW analysis. Of note, not all trained and certified i-BEST-1 sites enrolled patients to the study and therefore did not submit any LCI data before the study was stopped early (11 of 15 sites certified submitted LCI data). Therefore, the quality evaluation of data did not represent all sites trained and certified in this programme. However, the finding that 42% of tests were excluded, highlighted that it is important for sites and sponsors delivering MBW in clinical trials to plan strategies to facilitate repeat MBW tests in the case of invalid data and to consider how it might influence sample size calculations. Lower success rates (<50%) in CF studies have been explained mainly by issues relating to testing in children (inability to cooperate/tolerate the test) [15, 22]. In these studies, including older adults we have found that tolerability is still an issue contributing towards test exclusions (some test leak and not meeting end of test criteria due to intolerability) and indeed longer test durations would also impact on this.

Limitations

This study has a number of limitations, including that is was observational in design. Furthermore, this is the first time that this model of training, certification and over-reading has been delivered in bronchiectasis by the Belfast site and learning from centres in other disease groups will help inform planning for any future studies. Analysis of site success rates and the potential influencing factors in this study were exploratory and future studies would benefit from more rigorous analysis such as logistic regression to determine the significant factors impacting MBW success. While our study reported the main barriers to certification, it would also have been useful to collect reports from site operators on barriers to quality data post-certification. These qualitative data would be useful to interpret alongside the reasons for test exclusion, as we have reported. Furthermore, our analysis was focused on the operational aspects of MBW delivery. Further analysis on the MBW data (e.g. LCI results, test duration) could uncover other patient-related factors that could influence MBW feasibility in this patient group. Alongside the assessment of feasibility of LCI in this setting, determination of the clinimetric properties including long-term reliability and responsiveness of LCI are essential to inform a decision on the use of this outcome measure in future bronchiectasis studies.

Conclusion

In conclusion, this study demonstrates that MBW testing to measure LCI is feasible in a bronchiectasis multicentre clinical trial setting; however, key challenges remain regarding time to site certification and in obtaining the optimal proportion of valid data. This study highlights that the steps of training, certification and quality improvement are essential components to consider at the trial design and site feasibility stage. Future studies including LCI in the bronchiectasis patient population may wish to consider site experience in terms of training needs and ensure time for naïve sites to acquire experience, assess skill drift and the need for re-training during the study. Consideration of the patient cohort lung function and test duration may be important to estimate the percentage of quality data. Finally, the time, resource and capacity required for this activity is an important consideration for both sites and sponsors and could be informed by the experiences in this study.

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Appendix 1 00363-2019.supplement1

Appendix 2 00363-2019.supplement2

Appendix 3 00363-2019.supplement3

Appendix 4 00363-2019.supplement4

Appendix 5 00363-2019.supplement5

Appendix 6 00363-2019.supplement6

Appendix 7 00363-2019.supplement7

Appendix 8 00363-2019.supplement8