Chest computed tomography outcomes in a randomized clinical trial in cystic fibrosis: Lessons learned from the first ataluren phase 3 study.

A phase 3 randomized double blind controlled, trial in 238 people with cystic fibrosis (CF) and at least one nonsense mutation (nmCF) investigated the effect of ataluren on FEV1. The study was of 48 weeks duration and failed to meet its primary endpoint. Unexpectedly, while FEV1 declined, chest computed tomography (CT) scores using the Brody-II score as secondary outcome measures did not show progression in the placebo group. Based on this observation it was concluded that the role of CT scans in CF randomized clinical trials was limited. However, more sensitive scoring systems were developed over the last decade warranting a reanalysis of this unique dataset. The aim of our study was to reanalyse all chest CT scans, obtained in the ataluren phase 3 study, using 2 independent scoring systems to characterize structural lung disease in this cohort and to compare progression of structural lung disease over the 48 weeks between treatment arms. 391 study CT scans from 210 patients were reanalysed in random order by 2 independent observers using the CF-CT and Perth-Rotterdam Annotated Grid Morphometric Analysis for CF (PRAGMA-CF) scoring systems. CF-CT and PRAGMA-CF subscores were expressed as %maximal score and %total lung volume, respectively. PRAGMA-CF subscores %Disease (p = 0.008) and %Mucus Plugging (p = 0.029) progressed over 48 weeks. CF-CT subscores did not show progression. There was no difference in progression of structural lung disease between treatment arm and placebo independent of tobramycin use. PRAGMA-CF Chest CT scores can be used as an outcome measure to study the effect of potential disease modifying drugs in CF on lung structure.

RCT Entities:   Population Interventions Outcomes

Introduction

Outcome measures derived from chest computerized tomography (CT) are considered a more sensitive alternative for forced expiratory volume in 1 second (FEV1) to monitor structural lung changes [1-3]. To date only few large randomized controlled trials (RCTs) in cystic fibrosis (CF) have been conducted that included chest CT as an outcome measure [4, 5].

One of these studies was a RCT comparing ataluren to placebo, which was conducted from 2009 into 2011 [5]. This was a randomized, double-blind, placebo-controlled phase 3 study that included 238 CF patients, 6 years and older with at least one nonsense mutation (nm). The primary endpoint of this study was relative change in % predicted FEV1 from baseline to week 48 assessed by spirometry; the secondary endpoint was the rate of pulmonary exacerbations. The remainder of the endpoints presented were tertiary or exploratory. Change in lung CT scores were included as one of the exploratory outcome measures. The Brody-II CT scores were used as image analysis method [6]. The results of this study showed that there were no statistical significant differences between treatment arms in the primary outcome measure (FEV1) and secondary and tertiary outcomes including CT scores. In both the active and placebo groups %predicted FEV1 declined from baseline over the 48 weeks study duration. However, a post hoc analysis of the subgroup of patients not using chronic inhaled tobramycin showed a significant 5–7% difference in relative change from baseline in % predicted FEV1 and fewer exacerbations in ataluren treated patients relative to the placebo group at week 48 [5]. Based on this post hoc analysis it was concluded that treatment with ataluren might be beneficial for nmCF patients not receiving chronic inhaled tobramycin as tobramycin may interfere with the ribosomal binding mechanism of ataluren. For this reason, a second phase 3 study was conducted including only nmCF patients not treated with inhaled tobramycin using % predicted FEV1 as primary endpoint. In this subsequent study chest CT scans were no longer included as outcome measure (clinicaltrials.gov: NCT02139306).

It was surprising that despite the large number of CF patients with severe mutations in the first phase 3 ataluren study there was no disease progression over 48 weeks study duration as determined by the Brody–II CT scores. The authors of the study concluded in the online supplement that ‘the role of chest CT scans in study designs for CF clinical trials was limited’. This finding is in contrast to multiple smaller observational cohort studies, which have consistently demonstrated progression of CT scores over one- to two-year intervals and that CT scores are more sensitive than FEV1 as a marker of progression of structural lung disease [1, 3]. Unfortunately, details on the image analysis of this unique set of CT scans have not been published.

Since the completion and analysis of the first phase 3 ataluren RCT, two potential more sensitive and better standardized scoring systems for chest CT scans have been developed. The first is the CF-CT scoring system, which is an upgraded version of the Brody-II scoring system used for scoring of the CT scans in the first phase 3 ataluren study. The CF-CT scoring system uses well standardized definitions, reference images and standardized training of observers [7]. The second system is Perth-Rotterdam Annotated Grid Morphometric Analysis for CF (PRAGMA-CF) which is based on a morphometric approach and which has been shown to be more sensitive for the quantification of early CF lung disease relative to the Brody-II scoring and CF-CT scoring system [8]. Based on the availability of these two improved CT scoring systems and importance of the unique large ataluren chest CT scan dataset for CF drug studies, we repeated the CT scan analysis.

The aim of our study was to reanalyse all first phase 3 ataluren chest CT scans using both the CF-CT and PRAGMA-CF as independent scoring systems to phenotype structural lung disease in nmCF patients and to compare disease progression of structural lung disease over the 48 weeks between treatment arms. We also did a post hoc analysis to investigate whether our more sensitive CT scan analysis would have given useful information to the effectiveness of ataluren in nmCF patients with or without tobramycin treatment. We hypothesized that significant progression of structural lung disease can be observed in the study cohort using CF-CT and PRAGMA-CF scoring systems.

Methods

Study population

The randomised, double-blind, placebo-controlled, phase 3 trial of ataluren was performed between August 2009 and November 2011 at 36 sites in 11 countries in North-America and Europe [5]. Most important inclusion criteria included: age ≥6 years; abnormal nasal potential difference; sweat chloride >40 mmol/L; documentation of the presence of a nonsense mutation in at least one allele of the CF Transmembrane Conductance Regulator gene [5]. Inclusion and exclusion criteria of the ataluren study are detailed in the 1.1 in S1 File.

The protocol of the phase 3 ataluren trial (ClinicalTrials.gov, number NCT00803205) was reviewed and approved by the ethics committee or institutional review board of each participating institution. Written informed consent was obtained from patients or their custodians prior to patient screening. External oversight of the study was provided by a Study Steering Committee and an independent Data Monitoring Committee. The reanalysis of CTs was executed on coded CTs.

Chest CT scan

An inspiratory spiral low dose chest CT scan was obtained at the start of study (SOS) and end of study (EOS) in the ataluren phase 3 study in 210 of the 238 patients with nmCF. CT scanner specifics can be found in S1 Table in S1 File. The reason why 28 patients did not contribute a SOS and EOS CT was not defined in the paper by Kerem et al. and this information was not available in the data files supplied to us by PTC.

CT Scoring

We received 420 de-identified SOS and EOS CT scans from PTC of which 391 CT scans were of sufficient quality to be scored in random order using the CF-CT [7] and PRAGMA-CF [8] scoring systems. Each scoring system was performed by a single certified observer and both had 2 years of image analysis experience. Details on the training of the observers are given in 1.2 in S1 File.

The CF-CT scoring system evaluates the five lung lobes and lingua for severity and extent of central and peripheral CT abnormalities through eye balling. The following sub-scores were assessed: bronchiectasis, airway wall thickness (AWT), mucus plugging, and parenchymal disease. Each subscore is scored for each lobe on the inspiratory scan as absent, occupying 0–33%, 33–66%, or >66% of the lobe, and next multipliers are applied. Resulting scores range from 0 to 72, 0 to 54, 0 to 36, and 0 to 54 for these 4 components, respectively. The maximal possible total CF-CT score summing these subscores is 216 points. For statistical analysis CF-CT subscores are expressed as % of maximal score. The CF-CT scoring system takes around 30 minutes per CT scan.

PRAGMA-CF is a morphometric system that computes the volume fraction of structural lung components using a grid overlaying 10 equally spaced axial CT slices between lung apex and base. Each grid cell is scored with the following hierarchical system (highest to lowest priority): bronchiectasis, mucus plugging, AWT, atelectasis, and normal lung structure. PRAGMA-CF subscores are expressed as % total lung volume. The composite score %Disease for CF-CT and for PRAGMA-CF was computed, which is defined as the sum of %Bronchiectasis, %Mucus Plugging, and %AWT. To compute changes (Δ) in subscores over the study period, SOS values were subtracted from EOS values of patients with two available CT scans. The PRAGMA-CF scoring system takes between 30–40 minutes per CT scan.

Regions of low attenuation were not assessed as most (95%) patients had only three expiratory slices available [9]. To calculate intra-observer variability, a random subset of 20 CT scans was rescored after a one-month interval by both observers.

Statistics

Continuous variables are presented as mean and standard deviation (SD).

Intra-observer reliability for CT scoring was assessed using the intra-class correlation coefficient (ICC), a two-way mixed-effects model. Because no universally applicable standards are available for reliability, an ICC between 0 and 0.4 was considered poor, between 0.4 and 0.6 moderate, between 0.6 and 0.8 good and greater than 0.8 excellent.

We performed regression models and obtained the Pearson correlation (r) to investigate the association between PRAGMA-CF %Bronchiectasis with CF-CT %Bronchiectasis, PRAGMA-CF %Mucus Plugging with CF-CT %Mucus Plugging, and between PRAGMA-CF %Disease with CF-CT %Disease separately at SOS and EOS.

Bland Altman plots were generated plotting the difference between PRAGMA-CF and CF-CT subscores versus their mean values at both SOS and EOS. This was done for %Bronchiectasis, %Disease, and %Mucus Plugging being the subscores showing a significant change from SOS to EOS. Since the scale of the two methods is different we first standardized the variables.

Linear mixed effects models [10] were used to assess progression and compare effects of treatment for the following CT outcomes between SOS and EOS: PRAGMA-CF %Bronchiectasis, PRAGMA-CF %Mucus Plugging, PRAGMA-CF %AWT, PRAGMA-CF %Disease, CF-CT %Bronchiectasis, CF-CT %Mucus Plugging, CF-CT %AWT, CF-CT %Disease, and FEV1%predicted. Confounders included in the mixed-effects models were: Time: SOS/EOS, tobramycin treatment: yes/no and treatment group: ataluren/placebo. These models take into account the unbalanced nature of the data and that measurements from the same patients may be more correlated than measurements from different patients. In particular, they consist of two parts, namely the fixed part and the random part. The fixed-effects part describes the average evolution in time of a specific clinical parameter under study (e.g. PRAGMA-CF %Bronchiectasis), where this average is taken overall from the subjects in the sample at hand and is an estimate of the evolution of the clinical parameter in the target population. The random-effects (patient specific) part describes the evolution in time for each of the patients under study. This part accounts for the correlation in the data within patients.

For some outcomes when performing a mixed-effects and regression analysis we used transformations because the assumption that the variance of the error terms is constant (homoscedasticity) was not satisfied in the original scale. In particular, the square root transformation was used for the outcomes: PRAGMA-CF %Bronchiectasis, PRAGMA-CF %Mucus Plugging, PRAGMA-CF %AWT, and PRAGMA-CF %Disease in the mixed-effects models. Furthermore, the same transformation was performed for the %Bronchiectasis, %Mucus Plugging, and %Disease in the regression analysis for the comparison between PRAGMA-CF and CF-CT in the regression models.

We finally performed a simulation study using our PTC analysis results to estimate for future intervention studies the power assuming different scenarios. We performed these simulations for %Disease, and for %Mucus Plugging being potential reversible components of CF lung disease to obtain the power when assuming a 10, 30, and 50% reduction in progression from SOS will be obtained when treatment is initiated (more details can be found in the 1.3 and S5 Table: Power Analysis in S1 File).

No correction for multiple testing was performed. For all analysis R was used (version 4.0.2; nlme 3.1.148, lattice 0.20.41, lattice extra 0.6.29) [11]. For the ICC calculations we used IBM SPSS Statistics 21.

Results

The study population for this blinded reanalysis of 391 CT scans of the phase 3 ataluren RCT included 210 of the 238 ataluren study patients who contributed a SOS, and/or a EOS chest CT. A flow-chart of the study profile is shown in Fig 1. 29 out of the 420 chest CT scans we received could not be scored because of poor image quality, movement artefacts, or truncated image of the lung.

Fig 1

CT scoring was executed in parallel to the consecutive phase 3 ataluren study. Our image analysis was completed and database locked by January 2017. In March 2017 PTC communicated the negative results of the consecutive phase 3 study performed to validate the post-hoc group analysis results in those patients not treated with inhaled tobramycin as tobramycin was hypothesized to interfere with ataluren ribosomal binding mechanism of action. In 2018 after we obtained the relevant sections of the PTC study database which allowed us to execute an in-depth image analysis including comparing study arms. Patient characteristics of the CT study population at SOS are shown in Table 1. The image analysis data for CF-CT and PRAGMA-CF scores which were used for statistical analysis are shown in S1 Data.

Table 1

Patient characteristics at the start of study of patients with a Start of Study (SOS) and End of Study (EOS) CT which were included in the CT image analysis.

Patient characteristics at start of study (n = 195)
Gender
Male (%)	49.7
Female (%)	50.3
	Mean
Age (years)	22.7 (9.7)
Body weight (kg)	54.6 (13.3)
Body height (cm)	162.6 (12.6)
Body Mass Index (kg/m2)	20.3 (3.0)
FEV1 (l)	2.0 (0.7)
FEV1 (%Predicted)	61.3 (14.7)

Mean (± standard deviation).

CT scan analysis

The intra-ICC scores for the CF-CT scores were excellent for %Bronchiectasis, %Airway wall thickening, and %Mucus plugging and good for %Atelectasis/consolidations. The intra-ICC for the PRAGMA-CF scores were excellent for %Bronchiectasis and %Disease, good for Airway Wall Thickening, poor for atelectasis/consolidation and could not be computed for %Mucus Plugging because of too many 0 values. The exact ICCs are shown in S2 Table in S1 File. Bland Altman plots are presented in the S1A-S1C Fig in S1 File. Since the scale of the two methods is different we first standardized the variables.

Spectrum of disease at SOS and EOS

Results of mean CF-CT and PRAGMA-CF subscores at SOS and EOS for ataluren and placebo group were merged as there was no significant difference in scores at SOS and EOS between the two groups (Table 2). Results of mean CF-CT and PRAGMA-CF subscores at SOS and EOS for the ataluren and placebo group separately are shown in S3A and S3B Table in S1 File. Disease severity varied widely between patients as is illustrated in the stacked bar plots from SOS CT scans of this patient group for the CF-CT scores (Fig 2A) and PRAGMA-CF scores (Fig 2B). The mean CF-CT %Disease score at SOS for patients on ataluren was 17.65±6.58 (Range 0 to 37%). The mean PRAGMA-CF %Disease at SOS for patients on ataluren was 9.82±7.45 (Range 0 to 35.5%) (see Fig 2 and Table 2).

Table 2

This table shows the PRAGMA-CF and CF-CT subscores of CTs at the Start of Study (SOS) (n = 195) and End of Study (EOS) (n = 196).

Method	Subscore	SOS-CT	EOS-CT	delta	SD delta
CF-CT	%Bronchiectasis	15.03±9.05	16.28±9.67	1.25	0.95
CF-CT	%AWT	17.81±9.39	18.41±8.87	0.6	0.92
CF-CT	%Mucus plugging	29.05±10.67	29.29±11.25	0.24	1.11
CF-CT	%opacities	14.54±5.59	15.04±5.37	0.5	0.55
CF-CT	%Disease	17.94±6.71	18.67±7.03	0.73	0.7
PRAGMA-CF	%Bronchiectasis	7.79±6.56	8.86±7.48	1.07	0.71
PRAGMA-CF	%AWT	0.01±0.07	0.02±0.07	0.01	0.01
PRAGMA-CF	%ATL	0.12±0.18	0.13±0.26	0.01	0.02
PRAGMA-CF	%Mucus plugging	2.14±2.47	2.31±2.38	0.17	0.25
PRAGMA-CF	%Disease	9.96±7.58	11.19±8.45	1.23	0.81

Results shown are mean (± standard deviation). Delta = average difference between EOS and SOS values.

For the CF-CT scoring system subscores are expressed as % of maximum score, for PRAGMA-CF as the % of total lung volume. AWT = airway wall thickness, ATL = atelectasis.

Fig 2

Stacked bar plots showing the wide spectrum of disease scores found by the CF-CT (A) and PRAGMA-CF scoring system (B). A. This stacked bar plot shows the distribution of the CF-CT %total score scores at start of study (SOS) for 195 patients. Patients are sorted based on the CF-CT %total score. %total score for each patient is subdivided by the subscores %Bronchiectasis, %Mucus Plugging, %Airway Wall Thickening and %Atelectasis. Note the wide distribution of %total score ranging from 0 to 37%. B. This stacked bar plot shows the distribution of the % volume of the lung at start of study (SOS) for 195 patients occupied by structural lung abnormalities as determined by PRAGMA-CF score. Patients are sorted based on the PRAGMA-CF %Disease subscore which is the sum of PRAGMA-CF %Bronchiectasis, %Mucus Plugging, and %Airway Wall Thickening. %Atelectasis (is depicted on top of %disease). Note the wide distribution of %Disease severities ranging from 0 to 35.5%.

Correlation between CF-CT and PRAGMA-CF

Results of correlation between the mean CF-CT and PRAGMA-CF %Disease subscore at SOS and EOS CTs are shown in Fig 3. There was a clear and significant correlation for subscores between the two scoring systems. This was the case for SOS CT scans (Fig 3A) as well as for EOS CT scans (Fig 3B).

Fig 3

SOS (A) and EOS (B) CT scan image analysis results of %Disease subscore: CF-CT score vs. PRAGMA-CF method. This figure shows CF-CT %Disease scored by observer 1 plotted against PRAGMA-CF %Disease scored by observer 2 at the start of study (SOS, A) and at the end of study (EOS, B). The black line represents the regression line. The correlation between PRAGMA-CF and CF-CT for %Disease at SOS and EOS is 0.74 (p = <0.001).

Correlation coefficient and p values for SOS and EOS for %Disease were (r = 0.74 and r = 0.75 respectively, both p = <0.001) (Fig 3A and 3B). Similar correlations were observed at SOS and EOS for %Bronchiectasis (r = 0.71 and r = 0.70 respectively, both p = <0.001, S2A and S2B Fig in S1 File), and for %Mucus Plugging (r = 0.55 and r = 0.49 respectively, both p = <0.001, S3A and S3B Fig in S1 File).

Linear mixed models

CF-CT subscores did not show a significant progression from SOS to EOS. Progression for PRAGMA-CF subscores were found: square root of %Bronchiectasis showed a trend towards progression by 0.139 (p = 0.079) (Fig 4A). The square root of %Mucus Plugging increased significantly by 0.175 (p = 0.029), and for the square root of %Disease by 0.212 (p = 0.008) (Fig 4B). %AWT did not show a significant change.

Fig 4

Change of PRAGMA-CF %BE (A) and %Disease (B) from EOS to SOS per patient. These plots show the change in PRAGMA-CF %BE (A) and PRAGMA-CF %Disease (B) from start of study (SOS) to end of study (EOS). Grey open circles represent the measurements of individual patients and the grey lines are plotted between the measurements of a single patient at SOS and EOS time-point. Black solid line represents the change obtained by the linear mixed model for patients with ataluren and tobramycin treatment. Note that for other subgroups the lines would not be statistically different as there was no significant impact for any of the other investigated confounders. The dashed black lines represent the confidence interval, and the dashed red lines the prediction interval. Note that for PRAGMA-CF %Disease values are presented in the square root scale—which is the scale that was used to analyse the data as the errors for this subscore were not normal distributed. Furthermore, note that due to the large number of patients included in the figure the grey open circles might appear filled.

FEV1 did not show a significant change from SOS to EOS. There was no significant impact for ataluren and tobramycin treatment at baseline or over time on the linear mixed models results for %Bronchiectasis, %Mucus Plugging, %Disease and FEV1.

Table 3 shows the positive results of the linear mixed model analysis for PRAGMA-CF %Mucus Plugging and %Disease. All the other linear mixed model results for CF-CT subscores and %PRAGMA-CF subscores and FEV1% are shown in S4A and S4B Table in S1 File.

Table 3

Results of the linear mixed effects models for the outcome PRAGMA-CF %Mucus plugging and %Disease.

PRAGMA-CF %Mucus Plugging
	Value	Standard Error	p-value
(Intercept)	1.114	0.098	<0.0001
patient CT status (EOS)	0.175	0.080	0.0292
Treatment group (placebo)	0.124	0.132	0.3501
Tobramycin treatment (Yes)	0.226	0.154	0.1435
patient CT status (EOS): Treatment group (placebo)	-0.129	0.099	0.1910
patient CT status (EOS): Tobramycin treatment (Yes)	-0.116	0.100	0.2478
Treatment group (placebo): Tobramycin treatment (Yes)	-0.136	0.204	0.5048
PRAGMA-CF %Disease
	Value	Standard Error	p-value
(Intercept)	2.839	0.161	<0.0001
patient CT status (EOS)	0.212	0.079	0.0079
Treatment group (placebo)	-0.046	0.205	0.8240
Tobramycin treatment (Yes)	0.024	0.259	0.9251
patient CT status (EOS): Treatment group (placebo)	-0.075	0.098	0.4427
patient CT status (EOS): Tobramycin treatment (Yes)	0.020	0.099	0.8375
Treatment group (placebo): Tobramycin treatment (Yes)	0.156	0.358	0.6633

Confounders included in the models were: Time: SOS/EOS, Tobramycin treatment: yes/no and Treatment group: ataluren/placebo.

Power simulation study

Based on the PTC analysis when using PRAGMA-CF %Disease as the primary outcome measure in a new RCT 110 subjects per study arm would be needed to obtain a power of 0.9 assuming an overall reduction in progression of %Disease of 50% in the active treatment arm which is generally considered a clinical relevant reduction. When using PRAGMA-CF % mucus plugging as the primary outcome measure in a new RCT 170 subjects per study arm would be needed to obtain a power of 0.9 assuming an overall reduction in progression of %mucus plugging of 50% in the active treatment arm which is generally considered a clinical relevant reduction. Other data on the simulation analysis to obtain the power when assuming a 10, 30 and 50% reduction from SOS to EOS when treatment is initiated for PRAGMA-CF %Disease and %Mucus Plugging can be found in S5A and S5B Table in S1 File.

Discussion

In this study we have reanalysed a unique large set of chest CT scans obtained from 210 CF patients participating in the first phase 3 ataluren study. We used two independent chest CT scoring systems to evaluate the structural changes over a 48-weeks study period in the placebo and ataluren group with and without tobramycin treatment.

The most important observation is a significant increase in severity of structural lung disease according to PRAGMA-CF %Disease and %Mucus Plugging over the 48 weeks’ study period. Based on other CT scan analysis studies, progression of structural lung disease was to be expected in this cohort of CF patients with severe mutations. However, this finding is in contrast to the previous CT scan analysis performed for this study, when the Brody-II scoring system was used [5] which showed no significant changes in the Brody-II scores. There are a number of reasons to explain why we did observe disease progression and why this was not observed in the original analysis of the phase 3 RCT by PTC. The progression was observed for PRAGMA-CF subscores only and not for the CF-CT subscores. PRAGMA-CF was developed on morphometric principles which is a more sensitive scoring system especially for early lung disease in CF [12]. Our findings in this ataluren study demonstrate that PRAGMA-CF is also a more sensitive system than the CF-CT and the Brody-II scoring systems for more advanced disease in older patients. The PRAGMA-CF %Disease subscore was the most striking indicator of disease progression. This observation adds to the concept that %Disease subscore is an additional valuable tool to monitor changes in lung disease as it contains all relevant changes related to airways disease being %Bronchiectasis, %AWT, and %Mucus Plugging.

As we showed a highly significant correlation between PRAGMA-CF and CF-CT subscores at SOS and at EOS we would also have expected a significant increase of CF-CT subscores as well as CF-CT is considered to be more sensitive compared to the Brody-II system used for the first ataluren study by PTC. The CF-CT scoring system is based on the Brody-II scoring system which includes standardized training of observers by using an extensive instruction module with clear definitions, reference images, and training sets. The first reason why the CF-CT scoring system did not detect progression is probably the coarse scales on which abnormalities like bronchiectasis are scored. For example, bronchiectasis can be scored for each lobe as absent or as occupying 0–33%, 33–66%, or >66% of the lobe. Relevant but subtler (less than 33%) structural changes in the ataluren study might therefore have been missed. The second reason why the CF-CT scoring system did not detect any significant progression in the ataluren study is probably related to the relative short study duration of 48 weeks. To date the CF-CT scoring method has been primarily used in cohort studies for which biennial CT scans were analysed over longer periods of time than 48 weeks [3, 12, 13]. Overall, our findings are in support of PRAGMA-CF as a sensitive outcome measure in clinical studies to monitor structural changes in CF lung disease.

The second important observation of our study is that we did not find a significant difference in change in CT scores between SOS and EOS for the placebo or ataluren groups in the intention to treat (ITT) analysis. This was also the case comparing treatment arms for all patients not receiving inhaled tobramycin treatment. Hence, our post hoc-analysis contrasts with the reported post hoc-analysis from the first phase 3 ataluren study that showed higher % predicted FEV1 and fewer exacerbations in the ataluren group versus placebo in patients without inhaled tobramycin treatment.

In our analysis of 210 of the 238 study patients there was no longer significant improvement in FEV1 from SOS to EOS in the ataluren treated patients not treated with tobramycin versus placebo. Our CT scan analysis is therefore in concordance with the negative findings of the second phase 3 ataluren RCT which did not observe a significant treatment effect on FEV1 as outcome measure for ataluren treated patients versus placebo in patients naïve for tobramycin.

It has been well established in multiple observational studies that CT scores are more sensitive relative to FEV1 to track progression of structural lung disease in CF patients [1, 3]. In today’s CF care patients, using standard of treatment, have only little loss of functional outcomes such as FEV1 over one year [14]. For this reason, it has become important to include more sensitive chest CT outcome measures for RCTs of drugs that aim to reduce structural damage of the lungs. Hence, our results favour a multi-modality approach including functional and structural outcomes to understand whether a drug is effective. The addition of the PRAGMA-CF scan analyses, being a more sensitive CT scan measure, could have been valuable in the interpretation of the post hoc subgroup analysis and for the decision to run a further phase 3 study.

Substantial heterogeneity of structural changes in the lungs was observed in these 210 nmCF patients at baseline as is shown in Fig 2. A similar heterogeneity disease spectrum was observed in patients with moderate to end stage lung disease [6, 15]. In end stage lung disease the median %Disease was 29% with a range between 18–42%. Nonsense mutations are considered severe mutations which is confirmed by the severity of the structural changes that we observed in the ataluren population with a mean PRAGMA-CF % of 9.8% and values as high as 32%. The most important volume component of %Disease is %Bronchiectasis followed by %Mucus Plugging. The contribution of %AWT is small in contrast to early CF lung disease where this is the most frequent abnormality found [16]. This is the consequence of the hierarchical scoring order of the PRAGMA-CF scoring method. An airway that shows bronchiectasis or mucus plugging mostly will have a thickened wall but the first option for scoring is bronchiectasis, next is mucus plugging, and only when both are not apparent, is airway wall thickness assessed. Hence, this approach in patients with more advanced disease will result in an underestimation of airway wall thickness assessment. As %Disease includes all three relevant airway components it is not influenced by this hierarchical order.

Unfortunately, for this study we were not able to assess %Trapped Air as only 5% of patients had a volumetric expiratory scan which is needed for the accurate analysis of %Trapped Air [17].

For the development of more personalized treatments we suggest to include a chest CT scan in RCT at SOS to phenotype structural abnormalities of the study population and to improve our understanding of the response to treatment [18]. For example, PRAGMA-CF %Disease could be of use to predict the effect of an interventional drug on the clearance of mucus or airway wall thickness. Furthermore, the SOS CT scan gives us insight in the volume of structural changes that might still be reversible.

The results of this chest CT image analysis provides some guidance on the numbers of patients needed in a study that includes chest CT outcomes as it provides for the first time the 1-year natural history of CT change in CF [7, 8]. The significant increase of %Disease from SOS to EOS, observed in the current study, is largely the result of progression of %Mucus Plugging. The development of bronchiectasis is in general considered to be slow. It has been estimated that the ideal study duration for a disease modifying drug using %Bronchiectasis as outcome measure would be 2 years [19]. This PTC study supports the idea that for %Bronchiectasis a study duration of longer than 1 year is needed. Disease modifying drugs at best can stop progression of more advanced bronchiectasis, but is unlikely to reverse such bronchiectasis. This is not the case for PRAGMA-CF %Mucus Plugging which theoretically can be reversed for which reason a one-year study or even a shorter study duration of 3 to 6 months might be sufficient to measure its effect by CT scan analysis [20]. As PRAGMA-CF %Disease captures all changes related to airways disease it is a score that most likely can be improved over time. In addition %Disease would require only 110 subjects per study arm assuming a clinically relevant reduction of 50% in the active treatment arm versus placebo. This power can be substantially improved with standardizing CT scanning protocols, reconstruction techniques, and lung volume level during acquisition in clinical studies [18, 19, 21, 22]. This contributes importantly to the optimization of the image properties for image analysis balancing between radiation dose and image quality. This standardization is also important for reducing the number of CT scans that cannot be scored due to movement artefacts which was 7% for the PTC study. Furthermore, for PRAGMA-CF its sensitivity can be improved once it is automated. In addition, other promising automated image analysis systems are in development that are likely to be more sensitive to detect changes of airway dimensions [23]. For the development of such systems the PTC dataset is of great value for validation. Clearly for our power considerations one should keep in mind that we extrapolate from an observational trial scenario in mostly adult nmCF patients of a drug that was not effective to an interventional trial with a drug with a potential positive treatment effect. Ongoing studies in preschool children that are using chest CT as primary outcome measure will generate further evidence on the sensitivity of chest CT related outcomes in younger CF patients [18, 21].

In conclusion, progression of structural lung disease was observed in the 48 weeks ataluren nmCF RCT study cohort using PRAGMA-CF subscores. Importantly, in a post-hoc subgroup analysis of patients not treated with inhaled tobramycin, there was no significant treatment effect in ataluren treated patients on CT scan analysis outcomes using the PRAGMA-CF score. This analysis could have provided valuable additional data in assessing the validity of the positive treatment effect seen in FEV1 and exacerbation. We also show that a more extensive analysis of this unique set of chest CT scans of the ataluren study illustrates the important role that chest CT scan related outcomes can play in evaluating the effect of disease modifying drugs on CF lung disease in an RCT. These findings support the inclusion of chest CT scans in RCTs that aim to slow down progression or even reverse structural lung changes such as airway wall thickening related to CF lung disease.

(DOCX)

Click here for additional data file.

(XLSX)

Click here for additional data file.

16 Jun 2020

PONE-D-20-14479

Chest computed tomography outcomes in a randomized clinical trial in cystic fibrosis:

Lessons learned from the first ataluren phase 3 study

PLOS ONE

Dear Dr. Tiddens,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

This is an interesting and well-designed study. From the reviewer comments, you can see that they found merit in your study. Please find the careful comments enclosed, especially for some clarifications regarding data presentation.

Please submit your revised manuscript by Jul 31 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Sophie Yammine

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have provided information about ethics board approval and informed patient consent in the Ethics Statement. We ask that you additionally include this information in your Methods section.

3. We note that you add 'No - some restrictions will apply' on our data-shareing policy. Please clarify the nature of these restrictions, ie. If due to ethical or legal reasons.

In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

4. Thank you for stating the following financial disclosure:

This investigator initiated study was funded through an unconditional grant by PTC

Pharmaceutics, Inc. The corresponding author was given full access to all data that

was requested to PTC. Only data was within the scope of the study were requested.

The corresponding author had final responsibility for the decision to submit for

publication.

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

5. Thank you for stating the following in the Competing Interests:

The authors have declared that no competing interests exist.

We note that one or more of the authors have an affiliation to the commercial funders of this research study : PTC Therapeutics Inc.

a. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

6. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting and well-designed study that re-evaluated the CT scans acquired during a phase 3 randomized controlled trial (RCT) with two recent scoring systems that have been developed after the completion of that study.

I have only minor suggestions for the authors as detailed below to improve clarity and presentation.

Introduction:

1) Page 3, line 63. After detailing the primary and secondary endpoints, I would add a sentence to introduce tertiary endpoints: “The remainder of the endpoints presented were tertiary or exploratory. Chest CT was included ...”

2) Line 63. “Chest CT” might be better replaced with “Change in lung CT scores”

Methods:

3) Study population. Indicate that inclusion and exclusion criteria of the first PTC phase III study

are detailed in the supplement.

4) Chest CT scan. Lines 121-125. Figure 1 should be described in a specific paragraph at the beginning of the Results section. Please, avoid to cite figure 1 multiple times in the methods and results section, as it is confusing.

5) CT Scoring. Lines 128-129. Same as comment 4.

6) CF-CT scoring system requires more details: which is the score range for each subscore? How much time requires the scoring of each CT scan for both the CF-CT and the PRAGMA-CF scoring systems?

7) Lines 152-155. See comment 4.

8) Statistics. Could the authors re-order this paragraph according to the order of the results for clarity? (first ù intra-observer reliability, next correlation analysis, …

9) Line 176. Replace “of” with “between”

10) Line 188-189. Bland-Altman figures had to be cited in the results section, not here. Here cite only the method.

Results:

11) Start the results section with a paragraph describing the flow chart reported in Figure 1. Also, sorry but I don’t understand the flowchart:

a) At lines 210-211 you reported 211 patients (27 patients with no CT scans), while in the flowchart you reported 210 patients and 28 patients with no CTscan. Could you check the numbers?

b) In the flowchart, indicate that of the 238 patients, 120 were ataluren and 118 placebo?

c) 9 CT scans were incomplete: what do you intend? Also, 20 CT scans were not suitable for image analysis: why? (scanner parameters, artifacts…?)

d) 391 CT scans: (195 SOS+195 EOS?)

e) 195 CT scans: from patients trated with placebo (98 placebo SOS + 98 placebo EOS?)

f) 196 CT scans: from patients treated with ataluren (97 ataluren SOS+ 97 ataluren EOS?)

12) Table 1 (n=207?) Weren’t 211?

13) Lines 244-245. “The mean CF-CT % disease score for patients on ataluren was 17.65…” add “at SOS, and increased to 18.14…(p-value)” Report the p-value even if not significant.

14) Lines 245-246. Add mean PRAGMA-CF %disease at EOS and the p-value.

15) Table 2. Add column titles “method” and “subscore”.

16) Line 277. “subscores” might be substituted with: “%disease score”

17) Line 284. “S Fig 1A and 1B”, but you are referring to “S Fig 2A and 2B”. Please adjust the order of the supplemental figures according to their appearance in the main text.

18) Lines 282-288. “Pearson correlation=…, regression analysis…” might be better substituted by (r=…, p<0.001).

19) Line 287. “S Fig. 3A, 3B and 3C” but you are referring to S Fig 1A, 1B and 1C. Also, Bland Altman plot for % airway wall thickening is not reported.

20) Line 295 please rephrase as “r=0.74, p<0.0001”. Report n for the correlation analysis.

21) Line 298. Delete “significant” as % bronchiectasis shows a trend towards progression, but it is not significant.

22) Line 305. S Table 5A and 5B, but you refers to 4A and 4B.

Discussion:

23) Lines 360-361. Add the score range to the methods section.

24) Line 378. Remove “were”.

25) Line 380 In table 3 you have reported the results of the linear mixed effects models for the PRAGMA-CF subscores and not for FEV1

25) The discussion would benefit of a paragraph comparing the two scoring systems: training required by the radiologists, time to score each CT scan, effect of CT parameters and image artifacts on score.

26) What do you expect from the air trapping score? Wouldn’t be possible to quantify air trapping (low ventilation areas) only on inspiratory scans by thresholding images at -950 HU?

27) Could you comment on the additional dose related to the use of CT scoring system as outcome measure in a clinical trial?

Reviewer #2: In this article, Tiddens et al. reanalysed chest CT scans from patients with cystic fibrosis who participated in a phase 3 randomized control trial (RCT) of ataluren using newer CT scoring systems to determine if these would be more sensitive at detecting progression of disease. The original published results of the RCT did now show any improvement in FEV1, which was the primary endpoint, in patients who received the treatment and instead there was a decline in FEV1 % predicted in both groups. Other endpoints in this trial included chest CT scans measured at the start and at the end of study, where analysis with Brody II CT scores showed no difference between disease groups and no progression of disease at the end of the trial period. In the current study, the authors used two newer CT scoring systems and hypothesized that significant progression of structural lung disease can be observed in the study cohort using the newly developed CF-CT and PRAGMA-CF scoring systems. The authors provide a good rationale for conducting this study and the study is well-conducted. Therefore, my comments are mostly minor:

1. In the Statistics methods, it is simply stated that Linear mixed effects models were used for the outcomes. It would be helpful to state what the models aimed to achieve i.e. to show progression of CT outcomes, compare effects of treatment etc.

2. It would be good to briefly mention the results of the simulation study in the Results section, rather than in the Discussion, even if further detail is in the supplementary material. It should also be stated that power calculations were for PRAGMA outcomes.

3. In the first paragraph of the results section, the authors discuss analysis completed in January 2017 and results remaining unchanged in 2018 after obtaining the full PTC study database. It is unclear of the relevance of this information. Is the current analysis the results of the full study database?

4. The results for effect of treatment are not adequately described in the Results, but only shown in the Table 3. Since this was one of the aims of the study it should be highlighted in the Results section. Similarly, the results for Tobramycin warrants mention.

5. In the methods of the Linear mixed effects model it’s stated that FEV1%predicted is included in the model however the results for this are not shown.

5. Lines 238 to 240 of the Results section, the term ‘merged’ is used twice in the sentence.

6. Abbreviations such as CF and FEV1 are not defined on first use.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

17 Sep 2020

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

• As indicated we have made further changes to meets PLOS ONE's style requirements, including those for file naming.

2. We note that you have provided information about ethics board approval and informed patient consent in the Ethics Statement. We ask that you additionally include this information in your Methods section.

• As indicated we have now included in the methods section the same information about ethics board approval and informed patient consent as in our Ethics Statement.

3. We note that you add 'No - some restrictions will apply' on our data-sharing policy. Please clarify the nature of these restrictions, ie. If due to ethical or legal reasons.

In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

• As indicated we have now supplied underlying data used to reach the conclusions drawn in the manuscript and that is required to replicate the reported study findings in their entirety. A locked exel spreadsheet that contains all image analysis results of all CTs is now uploaded as part of the supplemental material. Upon request a code can be given to unlock the database. Access to the annotated CTs can be given within the Erasmus MC IT environment for legal / privacy reasons. Furthermore, depending on the question written permission by PTC for access to the CTs might be needed as PTS pharmaceuticals is the owner of the CT dataset.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

• As indicated we have followed the recommendations above and have added A locked exel spreadsheet that contains all image analysis results of all CTs is now uploaded as part of the supplemental material.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

• We have now shared the exell datasheet that contain the image analysis data of our CT analysis data on which the statistical analysis has been executed as defined in our methods section.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

4. Thank you for stating the following financial disclosure:

This investigator initiated study was funded through an unconditional grant by PTC

Pharmaceutics, Inc. The corresponding author was given full access to all data that

was requested to PTC. Only data was within the scope of the study were requested.

The corresponding author had final responsibility for the decision to submit for

publication.

Please state what role the funders took in the study. If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

• We have amended the funding and financial disclosure section as follows:

o Funding This investigator initiated study was funded through an unconditional grant by PTC Pharmaceutics, Inc. The corresponding author was given full access to all data that was requested to PTC. Only data within the scope of the study were requested. The corresponding author had final responsibility for the decision to submit for publication. The funder provided support in the form of salaries for EA, NB, JB, and MKvdC. Funder had no role in the image analysis study design and analysis, decision to publish.

o Conflicts of interest statement: JM worked at the time of the manuscript preparation at PTC and owns stock of PTC, he supplied us with relevant data when requested and he critically read the manuscript and gave relevant input where needed fulfilling all requirements for co-authorship. EK received a research grant through his hospital to perform the ataluren Phase III study and acted as Medical & Scientific Consulting Board member and received reimbursement for his time and for travel expenses. HT is director of the Erasmus MC LungAnalysis laboratory, he is a co-inventor of PRAGMA-CF which is patented. All financial aspects of the patent are handled by the Erasmus MC.

If this statement is not correct you must amend it as needed.

• See above

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

• As suggested we have added the funder statement and conflict of interest statement in the cover letter.

5. Thank you for stating the following in the Competing Interests:

The authors have declared that no competing interests exist.

We note that one or more of the authors have an affiliation to the commercial funders of this research study: PTC Therapeutics Inc.

Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

• As suggested we have adjusted the role of Joe McIntosh who at the time of the study was working at PTC. In addition we have reviewed and adjusted the author contributions. See our amended statements above.

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

• See our amended COI statement.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement:

"This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If this adherence statement is not accurate and there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

• As discussed: Access to the annotated CTs can be given within the Erasmus MC IT environment for legal / privacy reasons. Furthermore, depending on the question written permission by PTC for access to the CTs might be needed as PTS pharmaceuticals is the owner of the CT dataset.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

• We have as suggested included both an updated Funding Statement and Competing Interests Statement in the paper as well as in our cover letter

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

• In follow up on your comments we have scrutinized and updated all COI statements

6. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

• Changed according to guidelines

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

________________________________________

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

________________________________________

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

• We have now uploaded a study’s minimal underlying data set as Supporting Information files.

________________________________________

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

________________________________________

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting and well-designed study that re-evaluated the CT scans acquired during a phase 3 randomized controlled trial (RCT) with two recent scoring systems that have been developed after the completion of that study.

I have only minor suggestions for the authors as detailed below to improve clarity and presentation.

Introduction:

1) Page 3, line 63. After detailing the primary and secondary endpoints, I would add a sentence to introduce tertiary endpoints: “The remainder of the endpoints presented were tertiary or exploratory. Chest CT was included ...”

• We adjusted the sentence as suggested.

2) Line 63. “Chest CT” might be better replaced with “Change in lung CT scores”

We adjusted the sentence as suggested.

Methods:

3) Study population. Indicate that inclusion and exclusion criteria of the first PTC phase III study

are detailed in the supplement.

• We added this comment as suggested.

4) Chest CT scan. Lines 121-125. Figure 1 should be described in a specific paragraph at the beginning of the Results section. Please, avoid to cite figure 1 multiple times in the methods and results section, as it is confusing.

• We deleted the figure 1 citations from the methods section as suggested and mention Figure 1 only at the beginning of the results section.

5) CT Scoring. Lines 128-129. Same as comment 4.

• Changed as suggested, see above.

6) CF-CT scoring system requires more details: which is the score range for each subscore? How much time requires the scoring of each CT scan for both the CF-CT and the PRAGMA-CF scoring systems?

• We added the following requested information to this section: Each subscore is scored for each lobe on the inspiratory scan as absent, occupying 0-33%, 33-66%, or >66% of the lobe, and next multipliers are applied. Resulting scores range from 0 to 72, 0 to 54, 0 to 36, and 0 to 54 for these 4 components, respectively. The maximal possible total CF-CT score summing these subscores is 216 points. For statistical analysis CF-CT subscores are expressed as % of maximal score. The CF-CT scoring system takes around 30 minutes per CT scan.

7) Lines 152-155. See comment 4.

• Changed as suggested, see above

8) Statistics. Could the authors re-order this paragraph according to the order of the results for clarity? (first ù intra-observer reliability, next correlation analysis, …

• Changed as suggested

9) Line 176. Replace “of” with “between”

• Changed as suggested

10) Line 188-189. Bland-Altman figures had to be cited in the results section, not here. Here cite only the method.

• Changed as suggested

Results:

11) Start the results section with a paragraph describing the flow chart reported in Figure 1. Also, sorry but I don’t understand the flowchart:

a) At lines 210-211 you reported 211 patients (27 patients with no CT scans), while in the flowchart you reported 210 patients and 28 patients with no CTscan. Could you check the numbers?

b) In the flowchart, indicate that of the 238 patients, 120 were ataluren and 118 placebo?

c) 9 CT scans were incomplete: what do you intend? Also, 20 CT scans were not suitable for image analysis: why? (scanner parameters, artifacts…?)

d) 391 CT scans: (195 SOS+195 EOS?)

e) 195 CT scans: from patients treated with placebo (98 placebo SOS + 98 placebo EOS?)

f) 196 CT scans: from patients treated with ataluren (97 ataluren SOS+ 97 ataluren EOS?)

• We thank the reviewer for pointing out these mistakes. In addition the way we mixed numbers of CT scans and number of patients was inconsistent and confusing. We checked the numbers and they are now correct. We made changes to the flow chart (Fig 1) and text to make this more clear.

12) Table 1 (n=207?) Weren’t 211?

• Thank you for discovering this error. 211 is the correct number of patients from whom we were able to analyze 195 scans at baseline and 196 scans at follow up. 179 patients had both SOS and EOS, 16 patients had only a SOS and 17 EOS CT. We have now computed baseline characteristics only for 195 patients who had a SOS CT which we considered more correct.

13) Lines 244-245. “The mean CF-CT % disease score for patients on ataluren was 17.65…” add “at SOS, and increased to 18.14…(p-value)” Report the p-value even if not significant.

• In this paragraph entitled ‘Spectrum of disease at SOS and EOS’ we focus on the wide distribution of scores and not on the change. Therefore we did add ‘at SOS’ but preferred not to include the EOS values and the p values of the difference.

14) Lines 245-246. Add mean PRAGMA-CF %disease at EOS and the p-value.

• See answer on comment 13.

15) Table 2. Add column titles “method” and “subscore”.

• Changed as suggested

16) Line 277. “subscores” might be substituted with: “%disease score”

• Changed as suggested

17) Line 284. “S Fig 1A and 1B”, but you are referring to “S Fig 2A and 2B”. Please adjust the order of the supplemental figures according to their appearance in the main text.

• Sorry for this omission. We adjusted the order of the supplemental figures.

18) Lines 282-288. “Pearson correlation=…, regression analysis…” might be better substituted by (r=…, p<0.001).

• We adjusted this notation as suggested.

19) Line 287. “S Fig. 3A, 3B and 3C” but you are referring to S Fig 1A, 1B and 1C. Also, Bland Altman plot for % airway wall thickening is not reported.

• We excuse ourselves for this mix-up. We only showed the Bland Altman plots for the 3 outcomes that showed a significant change in the mixed models. We now clarify this in the text.

20) Line 295 please rephrase as “r=0.74, p<0.0001”. Report n for the correlation analysis.

• We adjusted this notation as suggested.

21) Line 298. Delete “significant” as % bronchiectasis shows a trend towards progression, but it is not significant.

• Deleted as suggested

22) Line 305. S Table 5A and 5B, but you refers to 4A and 4B.

• We excuse ourselves for this mix-up which thankfully was detected by the reviewer.

Discussion:

23) Lines 360-361. Add the score range to the methods section.

• As suggested we added the scoring ranges to the methods section.

24) Line 378. Remove “were”.

• Deleted as suggested

25) Line 380 In table 3 you have reported the results of the linear mixed effects models for the PRAGMA-CF subscores and not for FEV1

• Based on the comment by the reviewer we added the following sentences to the results section of the mixed models: ‘ FEV1 did not show a significant change from SOS to EOS. There was no significant impact for ataluren and tobramycin treatment at baseline and over time on the linear mixed models results for %Bronchiectasis, %Mucus Plugging, %Disease and FEV1.’ In addition ‘All the other linear mixed model results for CF-CT subscores and %PRAGMA-CF subscores and FEV1% are shown in S Table 4A and 4B’

25) The discussion would benefit of a paragraph comparing the two scoring systems: training required by the radiologists, time to score each CT scan, effect of CT parameters and image artifacts on score.

• We feel that it is beyond the scope of the manuscript to add more details on the scoring methods to the discussion section. We added information on the training for CF-CT and for PRAGMA-CF in the supplemental material section. We did not study the effect of CT parameters and image artifacts on the scores. However, we do mention in the discussion that 7% of CTs were excluded from analysis because of poor image quality. Standardization of CT protocols and lung volume will further improve the sensitivity of the scoring systems to track disease.

26) What do you expect from the air trapping score? Wouldn’t be possible to quantify air trapping (low ventilation areas) only on inspiratory scans by thresholding images at -950 HU?

• On the inspiratory scan low attenuation regions due to hypoperfusion in CF might be detected using thresholding. However, a spirometer controlled CT acquisition protocol is needed to obtain reliable data.

27) Could you comment on the additional dose related to the use of CT scoring system as outcome measure in a clinical trial?

• Routinely used low dose protocols can be used for scoring inspiratory scans and ultra-low dose for expiratory scans. This is risk benefit is discussed in reference 18 which we have now added to the standardization sections which we expanded as follows: ‘This power can be improved with standardizing CT scanning protocols, reconstruction techniques, and lung volume level during acquisition in clinical studies [18, 19, 21, 22]. This contributes importantly for the optimization of the image properties for image analysis balancing between radiation dose and image quality. This standardization is also important reducing the number of CT scans that cannot be scored due to movement artifacts which was 7% for the PTC study.’

Reviewer #2: In this article, Tiddens et al. reanalysed chest CT scans from patients with cystic fibrosis who participated in a phase 3 randomized control trial (RCT) of ataluren using newer CT scoring systems to determine if these would be more sensitive at detecting progression of disease. The original published results of the RCT did now show any improvement in FEV1, which was the primary endpoint, in patients who received the treatment and instead there was a decline in FEV1 % predicted in both groups. Other endpoints in this trial included chest CT scans measured at the start and at the end of study, where analysis with Brody II CT scores showed no difference between disease groups and no progression of disease at the end of the trial period. In the current study, the authors used two newer CT scoring systems and hypothesized that significant progression of structural lung disease can be observed in the study cohort using the newly developed CF-CT and PRAGMA-CF scoring systems. The authors provide a good rationale for conducting this study and the study is well-conducted. Therefore, my comments are mostly minor:

• We thank the reviewer for the time spend on our manuscript and for the comments which were helpful to improve the manuscript.

1. In the Statistics methods, it is simply stated that Linear mixed effects models were used for the outcomes. It would be helpful to state what the models aimed to achieve i.e. to show progression of CT outcomes, compare effects of treatment etc.

• As suggested we changed the sentence on mixed models to: Linear mixed effects models [10] were used to assess progression and compare effects of treatment for the following CT outcomes between SOS and EOS:

2. It would be good to briefly mention the results of the simulation study in the Results section, rather than in the Discussion, even if further detail is in the supplementary material. It should also be stated that power calculations were for PRAGMA outcomes.

• As suggested we now briefly included the key results of the simulation study for the PRAGMA-CF outcomes and the referral to Table 5A and 5B in the Results section and deleted these results from the discussion.

3. In the first paragraph of the results section, the authors discuss analysis completed in January 2017 and results remaining unchanged in 2018 after obtaining the full PTC study database. It is unclear of the relevance of this information. Is the current analysis the results of the full study database?

• We felt it relevant to mention this as it makes clear that we could not have been biased in our analysis by the results of the 2nd Phase III study. The reviewer is correct that the current analysis is based on the full study database which included for example the allocation to study arm and allowed us to investigate potential differences in outcomes between study arms. Bases on your question we have deleted this sentence. The section now reads as follows. ‘Our image analysis was completed and database locked by January 2017. March 2017, PTC communicated the the negative results of the consecutive phase 3 study performed to validate the post-hoc group analysis results in those patients not treated with inhaled tobramycin as tobramycin was hypothesized to interfere with ataluren ribosomal binding mechanism of action. In 2018 after we obtained the relevant sections of the PTC study database which allowed us to execute an in-depth analysis including comparing study arms.

4. The results for effect of treatment are not adequately described in the Results, but only shown in the Table 3. Since this was one of the aims of the study it should be highlighted in the Results section. Similarly, the results for Tobramycin warrants mention.

• We thank the reviewer for this useful comment. We focused the paper on our hypothesis ‘We hypothesized that significant progression of structural lung disease can be observed in the study cohort using CF-CT and PRAGMA-CF scoring systems.’ However, as was stated in the introduction: ‘We also did a post hoc analysis to investigate whether our more sensitive CT scan analysis would have given useful information to the effectiveness of ataluren in nmCF patients with or without tobramycin treatment.’ Hence we did not want to put to much emphasis on this post hoc analysis. Based on the comment by the reviewer we added the following sentences to the results section of the mixed models: ‘ FEV1 did not show a significant change from SOS to EOS. There was no significant impact for ataluren and tobramycin treatment at baseline and over time on the linear mixed models results for %Bronchiectasis, %Mucus Plugging, %Disease and FEV1.’

In the methods of the Linear mixed effects model it’s stated that FEV1%predicted is included in the model however the results for this are not shown.

• We have now added the following sentence to the results section of the linear mixed models: ‘FEV1 did not show a significant change from SOS to EOS. In addition we state that ‘All the other linear mixed model results for CF-CT subscores and %PRAGMA-CF subscores and FEV1% are shown in S Table 4A and 4B.’

5. Lines 238 to 240 of the Results section, the term ‘merged’ is used twice in the sentence.

• We deleted the first merged

6. Abbreviations such as CF and FEV1 are not defined on first use.

• We have introduce the abbreviations after first use.

________________________________________

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Rebuttal-20-08-2020_HT.docx

Click here for additional data file.

6 Oct 2020

Chest computed tomography outcomes in a randomized clinical trial in cystic fibrosis:

Lessons learned from the first ataluren phase 3 study

PONE-D-20-14479R1

Dear Dr. Tiddens,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Sophie Yammine

Academic Editor

PLOS ONE

.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

21 Oct 2020

PONE-D-20-14479R1

Chest computed tomography outcomes in a randomized clinical trial in cystic fibrosis: Lessons learned from the first ataluren phase 3 study

Dear Dr. Tiddens:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Sophie Yammine

Academic Editor

PLOS ONE