A comparison of clinic and home spirometry as longtudinal outcomes in cystic fibrosis.

BACKGROUND: The COVID-19 pandemic has accelerated the transition to telehealth, including the use of home spirometry in cystic fibrosis. Evaluating the accuracy and precision of longitudinal home spirometry is a requisite for telehealth-based research. This secondary analysis of a CF study (eICE) evaluates whether there are cross-sectional or longitudinal differences between home and clinic spirometry.
METHODS: Participants age ≥14 years with ppFEV1>25 were recruited from 2011-2015, issued a home spirometer, and asked to complete spirometry efforts twice per week for one year. Clinic spirometry was collected at baseline and every three months. Cross-sectional differences between clinic spirometry and the closest home spirometry measurement were analyzed. Longitudinally, we apply 5 methods to analyze the precision of home spirometry, and differences between clinic vs. home data.
RESULTS: Home spirometry is estimated to be 2.0 (95% CI: 0.3, 3.5) percentage points lower than clinic spirometry cross-sectionally. Longitudinally, the estimates of 12-month change in home spirometry varied by analysis method from -2.6 to -1.0 ppFEV1/ year, with precision markedly different. However, home spirometry change estimates were qualitatively similar to the clinic results: -3.0 ppFEV1/year (95% CI: -4.1, -1.9).
CONCLUSIONS: To leverage the potential cost, feasibility and convenience of home spirometry, the differences with clinic spirometry must be acknowledged. Significantly lower ppFEV1 in home devices shows that direct comparison to clinic spirometers may induce a spurious change from baseline, and additional variability in home devices impacts statistical power. The effect of coaching, setting, and equipment must be understood to use and improve home spirometry in CF.

Background

The emergence of SARS-CoV-2 infection leading to the COVID-19 pandemic has created a situation where clinic-based spirometry measurements are difficult to obtain [1,2]. Further, the recent approval of elexacaftor/tezacaftor/ivacaftor for up to 90% of persons with CF [3,4] may make clinic-based research visits less desirable in a CF population with newfound milder disease. There is both an acute and long term need to evaluate the use of more flexibly derived spirometry endpoints. Acutely, during the COVID-19 pandemic, design changes were necessary to ensure participant safety and adhere to institutional restrictions on clinical research. As a result, some ongoing studies incorporated home spirometry at follow-up visits, with the intention to compare against clinic spirometry collected at baseline. The effect of this change on accuracy and precision of treatment effect is unknown, but it could impact interpretation and regulatory evaluation. For new studies in development, that are considering home spirometry over the whole study period, an assessment of sample size and power using home devices is warranted.

The Early Intervention in Cystic Fibrosis Exacerbation (eICE) study was a one-year randomized, controlled trial in adolescents and adults with CF aimed at determining whether protocolized remote monitoring including patient reported outcomes (PROs) and home spirometry would lead to more rapid identification of acute pulmonary exacerbations [5]. It was hypothesized that earlier identification of pulmonary exacerbations would lead to benefits in pulmonary function. While enhanced monitoring was effective at more frequently identifying exacerbations, this did not translate into improved pulmonary function compared to the control group [6]. To date, the use of home or remote spirometry in CF clinical research has largely been as a clinical monitoring tool rather than a research endpoint [7].

We performed a secondary analysis of the eICE home and clinic lung function measures to determine if there are systematic differences or decreases in precision comparing home and clinic spirometry in a longitudinal trial. Additionally, several common analytic methods are contrasted to suggest strategies for optimizing the use of home spirometry data in future CF research.

Methods

Study administration

The study design for the eICE trial (NCT01104402) has been previously described [5]. From 2011 to 2015, 135 eligible study participants with age ≥14 years and forced expiratory volume in one second percent of predicted (ppFEV1) > 25 were randomized into the early intervention arm from 13 participating centers. Participants in the early intervention arm were issued an AM2+Ⓡ Lung Function Monitor (ERT, Inc.) [8] to be used twice weekly for 12 months. This analysis includes the 133 participants with at least one home spirometry measurement recorded. The device was designed and tested to meet American Thoracic Society standards [9]. Participants received an in-person demonstration of the home spirometry device from a research coordinator, technician, or other personnel (not necessarily the same personnel who collected clinic spirometry). Written instructions were also provided. No maintenance of home spirometry technique after the initial demonstration was attempted. Spirometry was to be measured in a seated position, after taking inhaled mediations and performing airway clearance techniques. Participants performed clinic spirometry on the standard of care schedule, once every 3 months over the study duration and in accordance with the 2005 ATS/ERS guidelines [9]. To encourage adherence, if participants had not uploaded a home spirometry measurement in 5 days they were contacted by the care center. Participation in the optional text message reminder system was low. While nose clips were used in the clinic per standard research practice, they were not used in the home setting. For each pulmonary function test (clinic or home), the highest value from a session was used in analysis. The 2012 Global Lung Initiative equations were used to calculate ppFEV1, which differs from the original eICE publication [6].

Statistical methods

Cross-sectional analyses evaluate pairwise differences (home-clinic) on nearby home and clinic observations for each participant. The nearest neighbor (NN) method pairs each clinic observation to a single home spirometry observation, specifically the nearest neighbor in time (limited to observations within 7 days). The windowed mean (WM) method pairs each clinic observation to the mean of all home spirometry measures within a ±7d window. Sensitivity analyses with greater window sizes are included for WM. Both methods utilize mixed models with a random intercept to account for the correlation of observations within each participant.

Longitudinal analyses (Fig. 1 ) evaluate the change from baseline and are applied independently to home and clinic data. NN and WM use the matching described above to select home spirometry observations, with change from baseline at each visit as the outcome, rather than the difference between home and clinic. The individual participant regressions (IPR) method first calculates individual linear regressions with time for each participant, then the slopes are analyzed across participants as derived variables. IPR is limited to subjects with at least 175 days of follow-up. The fourth method is mixed model, including a random subject intercept, and a linear relationship allowed for time (MMLT). The last method is a mixed model with a spline for time (MMST), specifically a natural cubic spline [11]. A quick reference for the methods is in Table 1 , longitudinal methods are diagrammed in Fig. 1, and additional details on each method are provided in the supplement.

Fig. 1

Illustration of all analytic methods on a single participant's home data. The gray line is the participant's home-measured ppFEV1. Clinic visit timing is represented by gray triangles along X-axis. Methods displayed are: NN (nearest neighbor; blue circles), WM (windowed mean; orange bar width indicates the window of home measures utilized), IPR (individual participant regression; green line), MMLT (mixed model linear in time; pink line), and MMST (mixed model with a spline for time; lavender curve). See supplement for additional details. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 1

Overview of the five analysis method abbreviations and whether they are included in cross sectional (Cross?) and longitudinal (Long?) analyses. For additional details see supplement.

Method	Description	Cross?	Long?
NN (Nearest Neighbor)	Each clinic measurement is matched to the home measurement closest in time for analysis.	Yes	Yes*
WM (Windowed Mean)	All the home spirometry observations in a window (±7d) around each clinic measure are averaged.	Yes	Yes*
IPR (Individual Participant Regressions)	A slope is derived for each participant with sufficient (175d) follow-up. The slopes are analyzed as derived variables to calculate the change from baseline.	No	Yes
MMLT (Mixed Model, Linear in Time)	A mixed model with a participant-level random intercept is fit using a linear model for time.	No	Yes
MMST (Mixed Model, Spline for Time)	A mixed model with a participant-level random intercept is fit using a natural spline for time.	No	Yes

* These analyses operate on a per-visit basis and thus the longitudinal analysis is really a change from baseline. Additionally, a maximum of one observation per clinic visit can be incorporated by them – or one window per visit for WM.

Regarding the selection of longitudinal methods, NN and MMLT are among the most common analytical approaches in cystic fibrosis trials. NN amounts to a paired t test at each visit, and MMLT is one of many models called a “mixed model for repeated measures”. WM and IPR have been proposed elsewhere as strategies which may be especially suited to home spirometry [10]. MMST adds some flexibility to MMLT, the primary motivation is that it may be suitable in a broader range of CF studies.

Statistical significance is defined by two-sided testing at the 0.05 level with no adjustment for multiple comparisons. Throughout we indicate 95% confidence intervals in parentheses, and all confidence intervals are derived using a nonparametric bootstrap with the percentile method. For any mixed model the bootstrap is resampled on the cluster (participant) level [12], and a note on the impact of this choice is included in the supplement. The R programming language [13] was used for all analyses, and a code tutorial is available online [14].

Results

Baseline characteristics and follow-up

The 133-participant cohort had a mean age of 26.5 years (SD = 11.5), and their mean baseline clinic ppFEV1 was 78.9% (SD = 22.0). 55 were F508del homozygous (41%) while 64 were F508del heterozygous (47%). Supplemental Table 1 contains additional baseline demographics as does the primary paper for the trial [6]. The median number of home spirometry measures was 56 per person. Adherence to the twice weekly home spirometry protocol decreased over the study period, with the number of home spirometry observations per subject-week being 1.81 at baseline, 1.19 in week 13, 1.13 in week 26, and 0.77 in week 39. Follow-up for clinic spirometry declined less sharply with 133 participants observed at baseline, 117 at the 13 week visit, 106 at the 26 week visit and 98 at the 39 week visit. A more complete description follow-up as it relates to the windows is available in Supplemental Table 2. While adherence decreased over the study, most subjects have some long-term home spirometry follow-up.

Cross sectional analysis

Fig. 2A displays the cross-sectional differences in ppFEV1, using all available data from the one-year study. Regardless of the method of estimation, a statistically significant difference exists between home and clinic spirometry. Using the NN method, the average within-participant difference between clinic spirometry and the nearest neighbor home spirometry observation is -2.01 (95% CI,-3.5, -0.3) ppFEV1, indicating that home spirometry is systematically less than clinic spirometry. Using WM, the magnitude of difference is greater with larger windows, down to -2.43 ppFEV1 with a 28d windowed mean. Fig. 2B shows the results of the NN analysis separated by visit to assess whether differences between home and clinic spirometry attenuate over time. The difference estimates remain negative at all visits, and there is insufficient evidence to show a systematic training effect over time. However, this analysis includes only those with all five time points (n=44) to eliminate censoring bias, so the sample available is small.

Fig. 2

A: Estimates of the cross-sectional difference using NN and WM methods, with WM repeated over various window widths (7d, 14d, …). The estimated difference, 95% confidence interval, and the standard deviation of the difference are labeled. B: NN analysis of each visit individually, limited to the subjects who have home and clinic data for all 5 visits (n = 44).

A difference versus mean plot (Bland-Altman, Fig. 3 ) using NN matching shows that some of the differences exceed 30 ppFEV1, and in the majority of outliers the home observation is greater than the clinic observation. Therefore, any post-hoc outlier exclusions would only enlarge the average cross-sectional difference. There was no significant association between spirometry differences and age or sex (Supplemental Figure 4). However, most of the extreme differences occurred in adolescents and young adults.

Fig. 3

Difference versus mean plot (Bland-Altman) for individual home/clinic observation pairs (matching as described for NN method). The mean difference and limits of agreement, ±1.96 times the standard deviation of the difference, are shown. The mean difference differs slightly from Fig. 2 as this is not a mixed model.

Longitudinal trajectory of lung function: clinic vs. home

The longitudinal results focus on the estimated change over time, rather than cross sectional difference. Estimates derived independently from home and clinic data are contrasted, along with how those estimates may differ depending on the analytical method applied. Fig. 4 shows ppFEV1 trajectories for home and clinic measures as a time-series in 4A and as 12-month change from baseline in 4B. The 12-month mean change (95% CI) using NN and home spirometry is -1.5 (-5.9, 3.0) ppFEV1, while the NN clinic data estimate is -2.9 (-5.1, -0.9). This shows the general trend of lower precision estimates using home data, compared with clinic data under all methods. With both home and clinic data the IPR, MMLT and MMST methods all have tighter confidence intervals, indicating superior precision compared to NN and WM. The estimates and precision are more method-dependent in home spirometry analyses, where the estimates ranged from -1.0 ppFEV1 in WM to -2.6 ppFEV1 in IPR, and the standard error of the change varies from 2.1 in NN to 1.0 in IPR. The longitudinal results, including the estimated 12-month change, depend heavily on both the data source and analysis method.

Fig. 4

A: Estimates of change from baseline using clinic spirometry measures (top row) and home measures (bottom row). Methods are shown with 95% confidence bands. B: Forest plot of the change from baseline estimated at 12 months, confidence intervals and standard errors for the change labeled at right. The confidence intervals in 4B are identical to the 12-month change estimates of each plot in 4A.

Discussion

There are important unanswered questions as to whether changes in pulmonary function captured via home spirometry measures are different from clinic spirometry, whether this difference changes over time, and if home spirometry results in a loss of precision compared to clinic spirometry. Our study begins to address these questions through a rigorous retrospective analysis of data from the eICE study. We have demonstrated a cross sectional difference between home and clinic spirometry in these data, and shown that the method of analysis can have a dramatic impact on the longitudinal analysis of home spirometry.

The existence of a cross sectional difference in ppFEV1 between home and clinic spirometry has been previously reported in patients with chronic diseases [15,16]. We have expanded on this work by confirming that a cross-sectional difference exists in a CF study population utilizing home spirometers. Additionally, we have shown that this difference does not resolve quickly over the course of a one-year study. A windowed mean was only modestly effective in reducing the variability of this difference, and it did not help at all with the bias. Larger windows exacerbated the cross-sectional bias. While other work generally supports the conclusion that home spirometry tends to be lower than clinic spirometry, the home versus clinic variability is sometimes lower than our results indicate [15]. Home spirometry was used as a monitoring tool in eICE, not a planned research outcome, and we cannot over-emphasize how important this context is in interpreting the results. Higher variability is likely a consequence of the study design. Specific contributors to lower variability in other work may include equipment (especially nose clips), data exclusions, coaching, testing of home devices in a clinic setting, or an unmeasured factor. Even two clinic spirometers used in the same setting will vary [17], so a certain portion of the difference is likely irreducible. Future work designed to parse the impact of each contributor will aid recommendations for counteracting cross-sectional bias.

Longitudinal evaluations of home spirometry have been conducted in related diseases and spirometric outcomes. Studies of forced vital capacity in idiopathic pulmonary fibrosis have pointed out that longitudinal outcomes may differ even if home and clinic spirometry are cross sectionally similar [18,19]. One of these studies concluded that the greater variability of home spirometry would require greater sample sizes to maintain power [19], while the other calculated that if the frequency of home spirometry is sufficiently high, that may yield superior longitudinal precision [18]. In a recent phase 3 randomized, placebo-controlled interventional trial in Duchenne muscular dystrophy (DMD) whereby both clinic and home spirometry were captured longitudinally over a year-long study, a retrospective study was performed to evaluate the comparability of treatment effect estimates derived from home vs. clinic-based peak expiratory flow [10]. Similar treatment effect treatment estimates were obtainable from remote spirometry and clinic spirometry. However, the analytical approach which produced treatment effect estimates closest to those obtained from the clinic spirometer suffered from a notable loss of precision. Our longitudinal analysis confirms the finding that home spirometry had lower longitudinal precision, and thus would require larger sample sizes to maintain power.

In our comparison of longitudinal analysis methods, we found that methods which model home spirometry over time (IPR, MMLT, or MMST) have superior performance compared to methods which analyze on a per-visit basis (NN or WM). This conclusion follows from the qualitatively similar longitudinal estimates using home data with IPR, MMLT or MMST, as compared to clinic data, and the superior precision compared to the nearest neighbor or windowed mean methods. Visit-based analyses are still pervasive for clinic spirometry, but the unstructured, frequent measurements in home spirometry make the difference between methods sharper. This finding agrees with previous work [18]. The magnitude of 12-month change in eICE was greater than in a typical CF study, but we would expect our findings to generalize to cohorts with a more modest ppFEV1 decline. While the IPR method is easily implemented and had good performance here, there are well known theoretical advantages to mixed models which properly account for within-subject correlation. As a result, we recommend the MMLT method as a good starting point for future analyses of home spirometry.

Home spirometry, particularly if unstructured observation timing or differential adherence exists, leads to a vastly different cluster size (observations per participant). Our use of cluster-level resampling is motivated by this observation, and we found that confidence interval method substantially impacts the results using home data but not clinic data (see supplement). For studies in this situation, we recommend a similar process to obtain confidence intervals. Home spirometry analyses which have a more rigid structure and high adherence, for example if the data collection was capped at once per week and adherence was high, may not need this additional complexity. In other words, it is more likely that the cluster size will vary in a home spirometry study design, but this is not a fundamental property to using home devices. Large outliers also contribute to the need for cluster-level bootstrap methods, which are likely reducible with more stringent real-time quality control.

While IPR and MMLT increased the precision of our longitudinal estimates, the imposition of a linear trend with time may not be universally suitable. This would be especially true for any study where participants experience an acute benefit from therapy, such as the recent trials of cystic fibrosis transmembrane conductance regulator (CFTR) modulators. Allowing for a nonlinear relationship with time has been shown to substantially impact studies of FEV1 in CF [20]. For cases when a nonlinear trend is likely, the MMST method, or any method which allows a nonlinear trend to be fit, will offer a more nuanced result, and decrease bias.

There are important challenges in home spirometry we have not addressed in this paper. COVID-19 has left many studies with a mix of home and clinic spirometry data, and we have not evaluated methods which attempt to model both concurrently. A simulation study, which would allow more precise notions of accuracy, precision, and coverage, may be appropriate to evaluate candidate methods.

There are several limitations with the analysis presented. First, adherence to home spirometry was low overall and it was lower than clinic visit adherence, which may have introduced bias related to missing data. Second, as noted above adherence waned over time making conclusions about longer term follow-up more challenging. Third, both technology and potentially the clinical stability of lung function has changed over time since this original study was conducted.

Home spirometry offers great promise in finding a safe, convenient, and cost-effective alternative to measuring pulmonary function in CF in the COVID-19 era and beyond. We provide cautionary results suggesting that assuming home and clinic spirometry are interchangeable may yield unsatisfying or misleading conclusions. Nonetheless, with appropriate administrative and analytical adaptations, home spirometry can become a useful tool in CF research.

CRediT authorship contribution statement

Alex Paynter: Writing – original draft, Visualization, Conceptualization, Formal analysis, Methodology, Software. Umer Khan: Writing – review & editing, Formal analysis, Validation, Software, Investigation, Methodology. Sonya L. Heltshe: Writing – review & editing, Formal analysis, Validation, Software, Investigation, Methodology. Christopher H. Goss: Writing – review & editing, Investigation, Funding acquisition. Noah Lechtzin: Writing – review & editing, Investigation, Funding acquisition. Nicole Mayer Hamblett: Writing – original draft, Investigation, Methodology, Conceptualization, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.