Comparison between an online self-administered and an interviewer-administered version of the Asthma Control Questionnaire: a cross-sectional validation study.

BACKGROUND: Online self-management programmes for asthma have recently become available. International guidelines suggest that the Asthma Control Questionnaire (ACQ) can be used in these programmes. In order to assess the current level of control and guide therapy, the same cut-off values are being used as in conventional asthma management. However, results might differ between different types of administration of the ACQ. AIMS: To assess the agreement between an online self-administered version of the ACQ and an interviewer-administered version at a routine visit.
METHODS: Cross-sectional data from primary care asthma patients in the Asthma Control Cost Utility Randomized Trial Evaluation (ACCURATE) trial aged 18-50 years and prescribed inhaled steroids were analysed. We selected patients who self-administered an ACQ online and subsequently had an ACQ completed by a nurse practitioner within 7 days at a trial-related control visit. ACQ scores were calculated and agreement assessed by paired t-tests, Pearson's correlation coefficient and a Bland-Altman plot.
RESULTS: A total of 351 patients were eligible (68% female, mean age 40 years). The time interval between the two versions was 3.2 days. There was a significant difference of 0.14 (95% CI 0.09 to 0.20; p<0.001) between the results of the online self-administered ACQ (mean 1.04±0.04) and the interviewer-administered ACQ results (0.90±0.04). The Pearson correlation coefficient was 0.79. The limits of agreement (-0.86, 1.14) exceeded the predefined minimal clinically important difference between results (±0.5). The Bland-Altman plot therefore showed insufficient agreement.
CONCLUSIONS: Assessment of asthma control by the ACQ is influenced by the type of administration. Our results suggest that better control of asthma is perceived when interacting with a caregiver than by online self-assessment.

Introduction

Online monitoring of asthma symptoms provides an opportunity of optimising patient-centred daily control. Online self-management programmes have therefore been developed which offer similar questionnaires to those being used in daily practice for assessment of asthma control. This policy is advocated in current guidelines which state that composite symptom scores such as the Asthma Control Questionnaire (ACQ) can be used in different settings by both patients and care providers to assess current asthma control.[1-4] As a consequence, even though the ACQ was originally developed and validated for guided self-administration, it is now administered in a variety of ways — for example, in self-management plans using pen and paper or internet applications,[5,6] self-administered but under guidance of a professional during regular control visits, or interviewer-administered by a practice nurse (PN), general practitioner or chest physician based on patients' responses.[1-3] Recent studies did not show statistically significant differences between paper and electronic versions of the ACQ[7] or between a postal-administered version and a version under guidance of a healthcare professional.[8] However, little is known about the agreement between online self-administered and interviewer-administered versions of the ACQ.

The aim of this study was to assess the agreement between an online self-administered version of the ACQ and an interviewer-administered ACQ by a PN during a routine control visit in adults with asthma.

Methods

Patients

The ACQ results of patients collected in the Asthma Control Cost Utility Randomized Trial Evaluation (ACCURATE) trial were used.[9] The trial was approved by the Medical Ethics Committee of the Leiden University Medical Center, Leiden, the Netherlands, and compares three different treatment strategies for asthma in primary care. Patients were included in the study if they were aged 18–50 years, had a doctor's diagnosis of asthma, and had received a prescription for inhaled corticosteroids in the previous year. Patients were excluded if they suffered from significant co-morbidity, if they were unable to understand written or oral Dutch instructions, or if they had been prescribed oral corticosteroids in the previous month.

Observations

In the ACCURATE trial, patients visited the general practice where ACQs were administered by a PN (interviewer-administered ACQ). Appointments were scheduled at approximately three-monthly intervals at a convenient time for both practices and participants. In addition, patients completed online questionnaires monthly at home, including the ACQ (online self-administered ACQ). Since the time between completion of the two versions of the ACQ was variable, we only selected observations from patients who self-administered an ACQ online and subsequently visited the PN for a trial-related control visit within 7 days. This time window was selected because it lies well within the advised range of 2–14 days for assessing reproducibility of instruments.[10] We excluded measurements where online self-administered ACQs were completed after the visit to the PN because trial-related medication changes might influence asthma control. We used a cross-sectional design to ensure that each participant was included only once.

Asthma Control Questionnaire (ACQ)

The ACQ consists of six questions on symptoms and one on pre-bronchodilator percentage predicted forced expiratory volume in one second (FEV1% predicted). In the present study, FEV1% predicted was assessed with a hand-held spirometer (PIKO-1) in the online self-administered ACQ and by routine spirometry in the interviewer-administered ACQ. In the online self-administered version, patients completed the questionnaire online at home without supervision, whereas in the interviewer-administered version the PN completed all questions based on patient responses. An overview of all methods of ACQ assessment is shown in Table 1. All questions were scored on a 7-point scale from 0 to 6 (where 0=good control, 6=poor control), and the composite ACQ7 score is the mean of the seven responses. The minimal important difference (MID) between results is 0.5.[11] Current control of asthma was divided into three levels: controlled (ACQ <0.75); partly controlled (0.75≥ACQ<1.5); uncontrolled (ACQ ≥1.5).[1,3]

Table 1

Different types of assessment of the Asthma Control Questionnaire

Statistical analysis

Paired t-tests were used to assess whether there were systematic differences between online self-administered and interviewer-administered ACQ results. The strength of the relationship was assessed using Pearson correlation coefficients. A Bland-Altman plot was used to assess the agreement between the two versions. In order to interpret a Bland-Altman plot, a clinically significant difference had to be predefined and results agreed sufficiently if they remained within this limit.[12] In the present study the MID of 0.5 was used for this purpose.

Linear regression and correlation analysis were performed to asses whether the time between self-administered and interviewer-administered questionnaires, age of participants, smoking behaviour, years of asthma history, or differences in sex between patients and PNs influenced ACQ results. All analyses were performed with STATA Version 11 (StataCorp LP).

Results

Of 611 patients participating in the ACCURATE trial, 351 visited the PN within one week after completion of the online questionnaire at home and were therefore eligible for the current analysis. Their mean age was 40.0 years, 68% were female, the mean inhaled corticosteroid dose was 861 μg beclomethasone equivalent, and 50% used a long-acting bronchodilator (Table 2). There was a statistically significant difference between the online self-administered (mean±SE 1.04±0.04, range 0–4.1) and interviewer-administered ACQ results (mean±SE 0.90±0.04, range 0–4.3) (difference 0.14; 95% CI 0.09 to 0.20; p<0.001). The Pearson correlation coefficient was 0.79.

Table 2

Baseline characteristics of participants in the current study and comparison with baseline characteristics of non-participants in the current study from the entire sample of 611 patients in the ACCURATE trial

When comparing the level of control, in 23% of patients the online self-administered version indicated a worse current level of control than the interviewer-administered version. This would lead to a step-up in maintenance treatment if the online self-administered ACQ were to be used to guide treatment decisions, which would not occur with the interviewer-administered ACQ. In 6% of patients the level of control was better, potentially triggering a step-down in medication. Overall kappa was 0.54, indicating moderate agreement (see Table 3 and Figure 1).

Table 3

Comparison between levels of control of the online self-administered and the interviewer-administered Asthma Control Questionnaire (ACQ)

Figure 1

Results of online self-administered Asthma Control Questionnaire (ACQ) results plotted against the results of the interviewer-administered version. On the x=y line of identity the results are identical and they agree sufficiently when they lie within the upper and lower minimal important difference (MID=0.5). The three categories of current asthma control (controlled, partly controlled, and uncontrolled) are also depicted. Treatment decisions will differ if the asthma control category by the online self-administered and the interviewer-administered versions of the ACQ do not concur

The Bland-Altman plot showed a mean difference of 0.14 between the two versions and the 95% limits of agreement (at ±1.96SD from the mean) were at −0.86 and 1.14 points. Both limits of agreement were beyond the predefined acceptable difference of (−)0.5 and therefore there was insufficient agreement between the two versions (Figure 2). Since the dispersion of the difference increased as the mean increased, we repeated the Bland-Altman analysis after log transformation and obtained similar results: mean log-transformed difference 0.16 (95% CI 0.10 to 0.22, p<0.001), 95% limits of agreement −0.98 and 1.31.

Figure 2

Bland-Altman plot showing the average scores of the online self-administered and interviewer-administered Asthma Control Questionnaire (ACQ) plotted on the x-axis against the difference between both results on the y-axis. The mean difference between the versions was 0.14. The predefined limits of sufficient agreement between both versions were the minimal important difference (MID) at +0.5 and −0.5 (dotted lines). The limits of agreement (±2SD) are also shown at 1.14 and −0.86 (solid lines). Both limits of agreement lie outside the MID and therefore there is not sufficient agreement between the two versions.

The difference between the two versions of the ACQ exceeded the MID of 0.5 in 25.1% of patients, of which 79.5% had a higher result in the online self-administered version and 20.5% in the interviewer-administered version. The differences were especially prominent in uncontrolled asthma. Furthermore, when questions in the ACQ were assessed separately, each question was significantly higher in the online self-administered version (p<0.03), except the question on FEV1 % predicted which showed no significant difference (p=0.39, Table 4).

Table 4

Comparison between results of the self-administered and interviewer-administered Asthma Control Questionnaire (ACQ)

The mean time between completion of the two versions of the ACQ was 3.2 days (95% CI 2.9 to 3.5). Linear regression and correlation analyses showed no significant association between the difference in ACQ scores and the time between visits (Pearson correlation coefficient (r)=0.03, p=0.54), age of participants (r=0.02, p=0.73), smoking behaviour (yes or no) (r=0.06, p=0.23), history of allergy (yes or no) (r=0.03, p=0.54), years of asthma history (r=0.04, p=0.52), and sex differences between patient and healthcare professional (r=0.01, p=0.81).

Discussion

Main findings

This study shows that the level of current asthma control is influenced by the type of administration of the instrument to assess control status. More severe symptoms were reported with the online self-administered questionnaire than with the interviewer-administered questionnaire by a PN based on patients' responses. This difference is of particular relevance due to the increasing use of questionnaires in clinical practice[13] and the appearance of online self-management programmes for asthma.[5,6] These programmes adjust therapy based on the same cut-off points of asthma control as those used in conventional asthma management, which may lead to different treatment advice in 29% (95% CI 24.6% to 34.1%) of cases.

Strengths and limitations of the study

Since we included patients from both rural and urban areas with the full range of uncontrolled to well-controlled asthma and less than 1% of all eligible patients for the ACCURATE trial were excluded due to exclusion criteria other than age, the strength of our study is that the 351 patients included comprise a representative sample of adult asthma patients aged 18–50 years. Also, the interviewer-administered ACQs were assessed by PNs in 119 different general practices so selection bias seems unlikely. A limitation of our study is that the results may perhaps not be applied straightforwardly to patients aged >50 years who were excluded because of the increased prevalence of chronic obstructive pulmonary disease and fixed airways obstruction in this group. Furthermore, online self-administered versions were always completed before the interviewer-administered version, which might have introduced an order bias. However, an analysis of 68 patients who self-administered an online ACQ within one week after a trial-related visit and in whom no trial-related change in asthma medication was prescribed confirmed the higher ACQ results in the online self-administered version (difference 0.17, 95% CI 0.04 to 0.29, p=0.01). The time difference between administrations of the two versions probably did not influence the results since the correlation analysis of difference in ACQ results and time between both administrations showed no significant association (r=0.03, p=0.54) (data not shown).

The online self-administered ACQ was used in a previous study[6] and validated in a preceding pilot study (which was not formally published). We showed that the repeatability was very good (intraclass correlation coefficient (ICC) 0.86) and the agreement was also very good (ICC 0.88). The obtained validation sample of subjects was a relatively homogeneous population with low mean scores for the ACQ (0.60) and no high results. Even though the ICC of 0.88 is slightly below the required 0.9 for validation,[3] we considered the online version validated since the ICC is highly dependent on between-subject variability. In our population sample the between-subject variation was relatively low and, therefore, the ICC of 0.88 underestimates the actual agreement. On the other hand, assessment by a PN based on patient responses is a non-validated method of assessment, albeit widely used. Therefore, in the present study we compared a validated method of assessment with a non-validated method of assessment which could (partly) explain the difference in results.

Interpretation of findings in relation to previously published work

Two previous studies compared a postal-administered version of the ACQ with a version under guidance of a healthcare professional,[8] and an electronic version with a paper self-administered version.[7] In both studies the ICC and Pearson's correlation were high and paired t-tests showed no statistically significant differences. Juniper et al. selected a more homogeneous group of only patients with uncontrolled asthma, whereas we included patients with the full range of asthma control. Furthermore, in the same study, Juniper et al. did find a significant difference between the electronic and paper versions of two other questionnaires with a more favourable outcome (i.e. fewer symptoms) with the paper version.[7] Also, in these studies, all assessments of the ACQ were completely self-administered while in our study it was completed by the PN based on patients' responses. Therefore, the actual person administering the questionnaire might explain the difference in ACQ results. In a review of different methods of administration of questionnaires, Bowling showed that patients reported more symptoms in self-administered than in interviewer-administered questionnaires in a number of fields of medicine, although not consistently.[14] For certain asthma symptoms, McDonald et al. recently showed that patients and healthcare professionals rate the importance differently.[15] Hence, the differences may be explained by a different interpretation of the question between patients and healthcare professionals, or PNs may weigh patients' answers and score their own interpretation. Finally, patients may express their symptoms more freely online or under-report symptoms when answering a PN due to social desirability, a known phenomenon when assessing highly sensitive personal behaviour.[16]

Several previous papers have shown that patients reported more symptoms in an online self-administered questionnaire than in a paper self-administered version,[14,17,18] suggesting that the format of an online questionnaire might have an effect. However, this was not a consistent finding,[14,18,19] and we also showed very good agreement between the online self-administered version and a paper self-administered version in our pilot study.

Implications for future research, policy and practice

Our results suggest that, during face-to-face contact with a healthcare professional, on average, symptoms of asthma are reported as less severe than with self-assessment of symptoms when the same questionnaire (i.e. the ACQ) is used. It remains to be seen if asthma is better controlled if guided by online self-administered (risking overtreatment) or by healthcare professional-administered measurement scales (risking undertreatment). In order to guide treatment by the level of asthma control, the type of assessment — even when using standardised tools — should therefore be taken into account.

Our study also underscores the relevance of validating new methods of assessment of questionnaires before using them in clinical practice.

Conclusions

The level of asthma control measured by the ACQ depends on the type of administration. Symptoms of asthma are reported as less severe in the interaction with a healthcare professional compared with online self-assessment at home.