Technical validity and usability of a novel smartphone-connected spirometry device for pediatric patients with asthma and cystic fibrosis.

BACKGROUND: Diagnosis and follow-up of respiratory diseases traditionally rely on pulmonary function tests (PFTs), which are currently performed in hospitals and require trained personnel. Smartphone-connected spirometers, like the Air Next spirometer, have been developed to aid in the home monitoring of patients with pulmonary disease. The aim of this study was to investigate the technical validity and usability of the Air Next spirometer in pediatric patients.
METHODS: Device variability was tested with a calibrated syringe. About 90 subjects, aged 6 to 16, were included in a prospective cohort study. Fifty-eight subjects performed conventional spirometry and subsequent Air Next spirometry. The bias and the limits of agreement between the measurements were calculated. Furthermore, subjects used the device for 28 days at home and completed a subject-satisfaction questionnaire at the end of the study period.
RESULTS: Interdevice variability was 2.8% and intradevice variability was 0.9%. The average difference between the Air Next and conventional spirometry was 40 mL for forced expiratory volume in 1 second (FEV1) and 3 mL for forced vital capacity (FVC). The limits of agreement were -270 mL and +352 mL for FEV1 and -403 mL and +397 mL for FVC. About 45% of FEV1 measurements and 41% of FVC measurements at home were acceptable and reproducible according to American Thoracic Society/European Respiratory Society criteria. Parents scored difficulty, usefulness, and reliability of the device 1.9, 3.5, and 3.8 out of 5, respectively.
CONCLUSION: The Air Next device shows validity for the measurement of FEV1 and FVC in a pediatric patient population.

INTRODUCTION

Diagnosis and longitudinal follow‐up of pulmonary diseases have relied on pulmonary function tests (PFTs) since the nineteenth century. Traditionally conducted in the clinic, spirometry can be a difficult technique, and the accuracy and repeatability depend on many factors such as equipment, patient effort, and supervision and encouragement of a technician. Nevertheless, a single PFT is no more than a snapshot of disease activity, and is unable to capture the variability of symptoms in chronic pulmonary disease.

Longitudinal data on a regular basis regarding pulmonary health could be very valuable for patients, clinicians, and clinical researchers, and this could be obtained by performing PFTs at the patients' home. An increase in readily available objective longitudinal data could be particularly useful in pediatrics, as children often find it difficult to perceive and express the severity of their symptoms. ,

Researchers have investigated the clinical value of home‐based measurements of several devices for pediatric asthma and cystic fibrosis (CF). While pulmonary outcomes were correlated to disease activity, the devices appeared to offer little benefit for clinical practice in terms of reduced admission rates, better disease control, or slower decline in pulmonary function. , , Since then, improvements in technology have allowed for the development of devices for measurement of complete flow‐volume curves at relatively low cost. An example is the Air Next spirometer, a Bluetooth connected device, allowing patients to perform spirometry tests with a smartphone. Use of the device has been reported in adult patients, but not yet in the pediatric population. , Before implementation in pediatric clinical care or clinical trials, a comprehensive technical validation of the device must be performed, consisting of the assessment of intra‐ and interdevice variability, comparison with conventional spirometry, as well as the assessment of usability for pediatric patients.

The aim of this study is to determine the agreement between the Air Next spirometer and conventional spirometry and to evaluate the usability of the device for children and parents when used at home.

MATERIALS AND METHODS

Location and ethics

This study was conducted at the Juliana Children's Hospital (HAGA teaching hospital, The Hague, The Netherlands) and Sophia Children's Hospital (Erasmus Medical Centre, Rotterdam, The Netherlands) from November 2018 to January 2020. The study protocol was reviewed and approved by the Medical Ethics Committee, Zuidwest Holland (The Hague, The Netherlands) before initiation of the study. The study was conducted according to the Dutch Act on Medical Research Involving Human Subjects (WMO) and in compliance with Good Clinical Practice. Written informed consent was obtained from all parents and children aged 12 years and older. Assent was obtained from children aged younger than 12. The trial was registered at the Dutch Trial Registry (NTR, Trial NL7611).

Subjects and study design

This analysis was part of a study investigating a novel home‐monitoring platform (CHDR MORE) in pediatrics. During this study, pediatric patients with controlled asthma (n = 30), uncontrolled asthma (n = 30), and CF (n = 30) were recruited from the outpatient clinic of the hospitals. All children were aged between 6 and 16 years. Asthma control was defined using the Global Initiative for Asthma criteria and Asthma Control Questionnaire (cutoff > 1.5 points). , Children and parents were given a 10‐minute training and practice session and were asked to perform PFTs once daily with the mobile device for a duration of 28 days. When logistically feasible, children visited the hospital to perform a conventional spirometry test at the outpatient clinic at the beginning or end of the study period and performed an Air Next spirometry test during the same visit. The sequence of tests was chosen based on preference for each patient.

Spirometry

Conventional spirometry was performed on a MasterScreen PFT (Vyaire, Mettawa, IL) at the Juliana Children's Hospital and the Sophia Children's Hospital, calibrated according to American Thoracic Society (ATS)/European Respiratory Society (ERS) guidelines. Home‐based spirometry was performed using the Air Next spirometry device (NuvoAir, Stockholm, Sweden). The device employs a turbine mechanism with disposable mouthpieces and cannot be calibrated by the user. The device uses Bluetooth to connect to a smartphone. Motorola G6 (Motorola, Chicago, IL) phones were used during the study. An accompanying application was installed, which uses age, sex, and height to calculate reference values according to the Global Lung Function Initiative 2012 equations, and requires Android 5.0 or higher. The application provides the forced expiratory volume in 1 second (FEV1), forced vital capacity (FVC), FEV1/FVC ratio, and peak expiratory flow (PEF) per maneuver.

Device variability

The Air Next device cannot be manually calibrated. We used a calibrated syringe (Viasys, Conshohocken, PA) with a capacity of 2994 mL to evaluate accuracy and the inter‐ and intradevice variability. The syringe was used to push the complete capacity through an Air Next device 20 times per device on 20 devices with a single turbine. In addition, the syringe was used on 25 different turbines with a single Air Next device.

Test procedures

ATS and ERS acceptability guidelines were used to judge and grade PFT quality (grade A‐F from best to worst). Spirometry maneuvers were acceptable if the start was rapid and without hesitation, the course of the expiratory maneuver was continuous, without any artefacts or evidence of coughing in the first second and if the end of the maneuver did not show early or abrupt interruption. The difference between the best two acceptable FVC and FEV1 should have been less than 150 mL. At least three maneuvers were performed per spirometry session. When it was difficult to obtain reproducible maneuvers during supervised measurements, a maximum of 10 maneuvers per patient were performed and the usable maneuvers were used. For home use, subjects were instructed to perform three maneuvers per session and were able to perform two additional measurements when appropriate (for example, mistiming of the forced exhalation or application errors). Subjects were not asked to self‐grade repeatability during the study period.

End‐of‐study questionnaire

At the end of the study period, a questionnaire regarding user experience was completed. Parents and participants were asked to give their opinion about the reliability of the device, the difficulty of using the device, and whether they found the use of the device to be useful or tedious on a 5‐point Likert scale.

Statistics

Baseline characteristics were summarized. Inter‐, intra‐, and turbine variability were calculated and expressed as a coefficient of variability (CV). Concordance between Air Next spirometry and conventional spirometry was assessed using the methods described by Altman and Bland. The mean differences between methods and the 95% limits of agreement were calculated for FEV1, FVC, PEF, and FEV1/FVC ratio. For FEV1 and FVC, acceptable bias was no more than 100 mL. For PEF and FEV1/FVC ratio, the acceptable average bias was 300 mL/s and 10%, respectively. , Pearson correlation coefficients between the two methods were calculated. Spirometry measurements at home were graded for quality and the number of maneuvers assigned to each grade were summarized descriptively. A mean grade per subject was calculated. The average mean grades of the three study groups were compared via a one‐way analysis of variance test and pairs were compared with Tukey's range test to adjust for multiple comparisons. Usability was evaluated by analyzing the end‐of‐study questionnaire completed by subjects and their parents. R version 3.5.1 was used for statistical analysis and visualization. Promasys software (OmniComm, Lauderdale, FL) was used for data management.

RESULTS

Baseline characteristics

A total of 90 subjects were included in the main study. The average age was 10 years (range, 6‐15). Subjects had performed an average of 12 (SD 11) hospital‐based PFTs before the study. Other baseline characteristics are displayed in Table 1.

Table 1

Baseline characteristics

	All participants (n = 90)	Comparison participants a (n = 58)
Age, mean (SD)	10.2 (2.7)	10.2 (2.7)
Sex
Male, n (%)	54 (60)	37 (65)
Female, n (%)	36 (40)	20 (35)
Diagnosis
Controlled asthma, n (%)	30 (33.3)	27 (47)
Uncontrolled asthma, n (%)	30 (33.3)	23 (40)
Cystic fibrosis, n (%)	30 (33.3)	7 (12)
Weight, kg, mean (SD)	39.5 (15.9)	40.8 (16.2)
Body mass index (SDS), mean (SD)	0.6 (1.4)	0.8 (1.4)
Height, cm, mean (SD)	144.1 (16.6)	144 (15.5)
Ethnicity
Caucasian, n (%)	69 (77)	37 (74)
Other, n (%)	21 (23)	15 (26)
Spirometry experience, n (SD)	12.2 (11)	8.4 (8)

Abbreviation: SDS, standard deviation score.

Comparison participants: patients who also performed conventional spirometry at the beginning or the end of the study period.

Of 400 measurements in 20 devices, the average bias from the calibrated 2994 mL was −40 mL (range, −124 to 56 mL). The average intradevice CV was 0.9% (range, 0.6%‐1.2%). Furthermore, the average interdevice CV was 2.8%. Average turbine bias was −70 mL and turbine CV was 1.8%. About 4% of measurements with the calibrated syringe exceeded the 3% accuracy threshold advised by ATS standards.

Measurement validity

Fifty‐eight subjects were able to perform hospital and Air Next PFTs subsequently. When comparing output between the two methods, there was one extreme outlier, most likely due to a technical defect resulting in a blockage of the outflow of the Air Next turbine, which was excluded from the statistical analysis. Figure 1 shows the limits of agreement and correlation between the Air Next and conventional spirometry of the several parameters. For FEV1, the average bias was 40 mL and the 95% limits of agreement were −270 and +352 mL. The Pearson correlation coefficient (R) was .97 (P < .001). The bias of FVC was 3 mL with limits of agreement of −403 mL and +397 mL (R = .97, P < .001). Furthermore, the analysis of PEF demonstrated an average difference of 590 mL/s (95% limits of agreement of −500 mL and 1690 mL) and the average difference for the FEV1/FVC ratio was 0.6% (95% limits of agreement of −5.8% and 7.0%). Although the correlation coefficient was lower as compared with FEV1 and FVC, there was still a good correlation between the two methods for both PEF (R = .93, P < .001)) and FEV1/FVC ratio (R = .91, P < .001). There was no proportional bias for any of the parameters. There was a correlation (R = −.33, P = .01 for FEV1 and R = −.26, P = .05 for FVC) between the absolute difference in FEV1 and FVC (expressed in % of predicted FEV1 and FVC) and age (Figure S1), but not between the absolute difference and previous spirometry experience, expressed as the amount of PFTs performed in the past (Figure S2). There was no statistically significant difference in absolute bias for FEV1 between the three groups (P = .28; Figure S3). When the absolute difference between the two methods was expressed as a percentage of the predicted FEV1 and FVC, the mean bias was 6.3% (SD 5%) of predicted FEV1 and 6.7% (SD 5.7%) of predicted FVC. The bias of FEV1 of subjects who performed the comparison at the end of the study period was slightly higher (3% of predicted, P = .009) compared to subjects who performed the comparison at the beginning of the study period (Figure S4).

Figure 1

Concordance between Air Next and conventional spirometry. A, C, E, and G, Bland‐Altman plots displaying the differences between conventional spirometry and Air Next spirometry against the averages of the two techniques for FEV1, FVC, FEV1/FVC ratio, and PEF, respectively. Dotted lines reflect the average bias (middle line) and the 95% limits of agreement (outer lines). B, D, F, and H, Pearson correlation between the two measurements. FEV1, forced expiratory volume in 1 second, FVC, forced vital capacity; PEF, peak expiratory flow

Technique and day‐to‐day variability

A total of 2047 spirometry measurements were performed with the Air Next device during the course of the study, resulting in an average compliance of 78%. The curves of 1821 sessions were available for analysis. When graded according to the ATS/ERS criteria, 45% of the FEV1 measurements were considered acceptable and reproducible, as well as 41% of the FVC measurements. A significant number of sessions were grade E, meaning they did not produce more than one acceptable maneuver or that the reproducibility was too low. About 2% of measurements were neither acceptable nor usable for both FEV1 and FVC. Summarized grades are listed in Figure 2A,B. There was a statistically significant difference on average grade between CF patients and patients with uncontrolled asthma (FEV1, P = .02; Figure 2C and FVC, P = .03; Figure 2D). Age and average grade were not correlated (Figure S5). Day‐to‐day CV of acceptable trials (grade A‐C) was 9.0% (SD 5.7%) for FEV1 and 7.7% (SD 5.4%) for FVC.

Figure 2

ERS/ATS grades for measurements performed at home. All spirometry sessions were graded according to ATS/ERS guidelines for FEV1 and FVC separately. Grade A‐E represent sessions with acceptable maneuvers but with varying repeatability. Grade U includes session with usable but not with acceptable maneuvers and grade F is reserved for session without acceptable or usable maneuvers. A, Proportion of spirometry sessions that were awarded each grade for FEV1. B, Proportion of spirometry sessions that were awarded each grade for FVC. C, Boxplot of average FEV1 grade per study group. Dots represent individual averages. There was a statistically significant difference between the CF and uncontrolled asthma group (P = .02). D, Boxplot of average FVC grade per study group. Dots represent individual averages. There was a statistically significant difference between the CF and uncontrolled asthma group (P = .03). ATS, American Thoracic Society; CF, cystic fibrosis; ERS, European Respiratory Society; FEV1, forced expiratory volume in 1 second, FVC, forced vital capacity [Color figure can be viewed at wileyonlinelibrary.com]

Usability

Sixty‐nine (77%) subjects completed the end‐of‐study questionnaire. In general, parents found the use of the spirometry device to be acceptable. When asked to score their agreement with the statement “I found the use of the spirometer to be tedious,” the average score was 1.8 out of 5 (SD 1.1). Furthermore, parents scored the difficulty 1.9 out of 5 (SD 1.2), usefulness 3.5 out of 5 (SD 0.9) and the perceived reliability 3.3 out of 5 (SD 1.0). Summarized results are displayed in Figure S6.

DISCUSSION

The current study investigates the technical validity and user experience of the Air Next spirometer for pediatric patients. Air Next spirometer output was compared with the gold standard: conventional spirometry in the clinic. Subjects and their parents also completed a questionnaire regarding the usability of the device.

The interdevice, intradevice, and turbine variability were assessed with a calibrated syringe of 2994 mL. All of the measurements were within 125 mL of the reference. Although 125 mL exceeds the 3% accuracy standard advised by the ATS, 96% of measurements fell within the 3% range. The coefficients of variability were all below 3%, which suggests that the repeatability of the device is good.

Bland‐Altman plots displaying the difference between the Air Next measurements and conventional spirometry demonstrated a negligible bias for FEV1, FVC, and FEV1/FVC ratio of 40 mL, 3 mL, and 0.6%, respectively. Furthermore, the 95% limits of agreement for FEV1 and FVC are comparable with earlier studies in adults. , Both FEV1/FVC ratio and PEF showed relatively wide limits of agreement compared with conventional spirometry, while PEF demonstrated bias compared with the gold standard. Interestingly, concordance of PEF was not reported in earlier publications. While the grades of the supervised spirometry sessions with the Air Next were all adequate (A‐C) according to ATS/ERS criteria, we suspect the individual differences of FEV1 and FVC measurements, and the consistently lower PEF of the Air Next measurements to be mainly due to differences in technique. Subjects had to coordinate several actions in quick succession: initiating the smartphone application, complete a full forced inspiration, perform a controlled arm movement towards the mouth, and finally complete a forced expiration. This is a relatively complex sequence of actions compared with conventional supervised spirometry and could influence the maximum effort given to the forced expiration. The complexity of the sequence of actions may also explain the correlation between absolute difference in FEV1 and age. For most subjects, the spirometry session for comparison was the first time they used the Air Next device. However, more familiarity with the technique did not appear to lead to better concordance, considering the observation that children who performed the comparison at the end of the study period did not exhibit a smaller deviation from conventional spirometry. We hypothesize this may be due to a decrease in motivation in children who performed daily PFTs during the preceding 28 days. Another important difference that may explain discordance is that small devices exert low resistance to expiration in comparison with conventional devices, which may affect the way children perform PFTs. While the bias of 0.59 L casts doubt on the absolute accuracy of the device for PEF measurements, the FVC, FEV1/FVC ratio, and especially FEV1 are considered to be more important parameters of pulmonary health. Furthermore, the measured PEF may show good correlation with symptom severity in the case of home monitoring. The limits of agreement for FEV1 and FVC are wider than the bias of the Air Next device determined with the calibrated syringe. This suggests that individual differences between the Air Next and conventional spirometry are the result of bias by both the patient and the device. A subgroup analysis of 25 children who displayed good technique in the home setting (median grade A‐B) showed slightly smaller limits of agreement. (Figure S7). Limits of agreement of this magnitude are inherent to direct comparisons of spirometers, as demonstrated by the literature on this subject. , , , Still, the relevance of the individual differences of this magnitude is higher in pediatrics, because their smaller expected lung volumes lead to biases that may be clinically relevant.

Subjects used the device at home for 28 consecutive days in the main study. Individual curves were assessed and graded according to the ERS/ATS criteria. The majority of measurements would be considered suitable for further analysis, but 36% of FEV1 measurements and 39% of FVC measurements were graded D, E, U, or F, meaning that they were not performed technically adequate. Interestingly, patients with uncontrolled asthma appeared to exhibit worse technique than patients with CF. A number of sessions with poor technique could have been the result of dyspnea due to the underlying disease, and the obtained values for FEV1, FVC, and PEF could still correlate well with perceived symptoms. However, the difference in technique could also be explained by the fact that children with CF perform a PFT every 3 months, which results in more familiarity with the technique. Therefore, this observation could also indicate a need for more training sessions, which has been reported to be beneficial for improving inhalation technique. Extensive training could be beneficial for home‐based spirometry as well and could be investigated further during a clinical validation study. Although the acceptability criteria that were the cause of a maneuver being unacceptable were not routinely recorded, the unacceptable maneuvers most often did not reach the end of forced expiration criteria. A high back‐extrapolation volume was encountered often as well. Both are indicators of insufficient effort during the end and start of the maneuver, respectively.

According to the end‐of‐study questionnaire, parents and children did not find the measurements to be difficult, although this assessment may change when immediate feedback on the quality of the measurements is provided. During the study, some participants had recurrent Bluetooth connectivity problems, which may be related to the used phone or the particular device that was used. To optimize reliability and usability, more intensive training and strict instructions may be necessary. During this study, participants underwent a 10‐minute training, which may not be enough to prevent wrong conduct. Still, issues such as low motivation, technological glitches, or even something as trivial as blocking air inflow with the tongue or air outflow with the hands are difficult to avoid completely without the supervision of a trained technician. This was demonstrated by the extreme outlier excluded in our analysis. Issues such as these may cause false positive or false negative results when used for the remote diagnosis of pulmonary obstruction.

Nevertheless, when correctly performed, the Air Next demonstrates reliability for FEV1 and FVC measurements compared with conventional spirometry and with a good user experience. In clinical care, the device could support home monitoring and provide timely information to patients when to contact a doctor. Furthermore, the device can be used for the purpose of telemedicine, which may be increasingly used during and after the crisis precipitated by the coronavirus disease‐2019 pandemic. Although previous studies indicating home‐based spirometry does not add value to pediatric clinical care, this may change when combined with other assessments, such as a symptom questionnaire, a wearable device, or other monitoring techniques. This may help physicians to improve monitoring of pediatric patients, while reducing the burden of disease. In addition, with the increasing popularity of digital endpoints and decentralized clinical trials, the device could play an important role in future clinical trials for pediatric CF, asthma, and other pulmonary diseases, which could decrease the burden of clinical trial participation. Finally, the device may be useful for primary care physicians without access to conventional spirometers in low‐income countries or rural areas, or at the point of care in patients' homes.

This study has some limitations, one of which is that not all of the participants could be included in the validation group. This is mainly due to logistical reasons and the fact that the comparison was part of a secondary analysis of a clinical study. However, there were no large differences in baseline characteristics between the complete cohort and the validation cohort (Table 1). The nonrandomized order of tests may have influenced the results through spirometry‐induced bronchoconstriction. However, we did not diagnose this condition in any of the included subjects. The curves of 226 spirometry sessions were unavailable for review due to application connectivity errors. However, this issue occurred at random and, therefore, did not impact our overall conclusions. Although we found no correlation between the absolute bias and previous spirometry experience when comparing conventional spirometry to the Air Next, the proportion of highly experienced subjects was low. A higher number of experienced subjects may have resulted in a better correlation. A strength of the study is the inclusion of pediatric patients with controlled asthma, uncontrolled asthma, and CF, giving a representative sample of possible pediatric target populations. The manufacturer has unlocked additional functions of the device since the initiation of this study, allowing for the measurement of the inspiratory measurements forced inspiratory vital capacity, peak inspiratory flow, maximal inspiratory flow, and maximum expiratory flow. These functionalities should be independently validated before integration in clinical care or clinical trials. Future clinical validation of home‐based measurements with the Air Next will be performed to determine the objectivity and reproducibility of longitudinal unsupervised measurements.

CONCLUSION

The Air Next spirometer is technically valid for the measurement of FEV1 and FVC in children aged 6 to 16, while PEF measurements show significant bias. The user experience was considered favorable by subjects and their parents. FEV1 and FVC measured at home could add significant value to clinical care and clinical trials, but future studies should determine the clinical value of home‐based spirometry measurements for the purpose of monitoring disease‐activity or response to treatment, possibly in combination with other home‐based measurements.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

AUTHOR CONTRIBUTIONS

MK conducted and designed the study, analyzed the data, and wrote the manuscript; EE conducted the study, analyzed the data, and reviewed the manuscript; NE performed measurements and reviewed the manuscript; HJ, IG, AS, and MN recruited patients and reviewed the manuscript; AZ supported data analysis; FS and AC designed the study and reviewed the manuscript; and GJD designed the study, supervised study conduct, and reviewed the manuscript.

Supporting information

Click here for additional data file.

Supporting information

Click here for additional data file.

Supporting information

Click here for additional data file.

Supporting information

Click here for additional data file.

Supporting information

Click here for additional data file.

Supporting information

Click here for additional data file.

Supporting information

Click here for additional data file.