Psychometric validation of the experience with allergic rhinitis nasal spray questionnaire.

BACKGROUND: Patient experience and preference are critical factors influencing compliance in patients with allergic rhinitis (AR) receiving intranasal corticosteroids. The Experience with Allergic Rhinitis Nasal Spray Questionnaire (EARNS-Q) was developed to measure subject experiences with and preferences for nasal sprays.
OBJECTIVE: To describe the psychometric validation of the EARNS-Q modules.
METHODS: An observational study was conducted with subjects aged 18-65 years with physician-diagnosed vasomotor, seasonal, and/or perennial allergic rhinitis who were using a prescription nasal spray. Subjects completed the experience module of the EARNS-Q and the Treatment Satisfaction Questionnaire with Medication (TSQM) at baseline and after 2 weeks. Further validation analyses were conducted in a 3-week, randomized, single-blind, crossover, multicenter clinical study in which subjects ≥18 years of age with documented seasonal AR received flunisolide and beclomethasone and completed the EARNS-Q experience module on days 1 and 8, the EARNS-Q preference module on day 22, and the TSQM on days 8 and 22.
RESULTS: The observational and clinical studies were completed by 121 and 89 subjects, respectively. Both modules demonstrated acceptable reliability (α = 0.72 experience module; α = 0.93 preference module global scores) and validity (intraclass correlation coefficient or ICC 0.64 to 0.82 test-retest validity). Correlations among the experience and preference modules were moderate (r = 0.39 to 0.79) and within internal consistency reliability estimates, indicating measurement of distinct constructs.
CONCLUSION: The EARNS-Q is a patient-reported outcomes measure that enables reliable and valid measurement of subject experience with, and preference for, prescription intranasal corticosteroid sprays for allergic rhinitis.

RCT Entities:   Population Interventions Outcomes

Introduction

Allergic rhinitis (AR) is an inflammatory disease that results from an immunologic response to an allergen in a sensitized individual. Binding of immunoglobulin E to the allergen stimulates the release of histamine, leukotrienes, and other inflammatory mediators. Allergic rhinitis is commonly characterized by sneezing, rhinorrhea, nasal congestion, nasal pruritus, and watery, itchy eyes. Allergic rhinitis may be classified as seasonal (typically triggered by pollen) or perennial (caused by animal dander or dust mites).1,2

Allergic rhinitis is highly prevalent, with recent estimates indicating that AR affects up to 40 million people in the US (approximately 10%–20% of the general population).1,3 The symptoms of AR often lead to impaired physical and social functioning, sleep disturbances, reduced quality of life, loss of concentration, impaired cognitive function, poor examination performance, decreased productivity, increased work loss, and increased medical costs.3–7

Pharmacologic options for the management of AR include intranasal corticosteroids, antihistamines, decongestants, cromolyns, antileukotrienes, ipratropium bromide, and omalizumab.8 Of these, intranasal corticosteroids are the most effective and therefore are considered first-line therapy for moderate-to-severe or persistent AR. Studies have shown that patient acceptance and preference are important factors in choosing the right intranasal corticosteroid treatment for AR.9,10

Patient preference for intranasal corticosteroids has been linked to greater expected compliance,11,12 and is thought to be determined by the experience a patient has with a particular treatment.13,14 As there is often a need for regular, long-term use of these medications, patient compliance may be influenced by the product’s sensory attributes, such as smell, taste, aftertaste, leaking out of the nose or down the throat, and the feel of the spray in the nose or throat.15

Questionnaires are commonly used to assess patient preference for intranasal corticosteroids.16–18 However, no available instrument also measures experience with medication. The Experience with Allergic Rhinitis Nasal Spray Questionnaire (EARNS-Q) incorporates many of the sensory attributes that subjects with AR have expressed as important in having a pleasant experience with nasal sprays. The reliability of the questionnaire in quantifying such sensory attributes can be described by its psychometric (statistical) properties.

The EARNS-Q consists of two modules: the experience module, which includes 28 items that assess experience with nasal sprays, and the preference module, which includes the experience module plus 15 items that assess preference for a nasal spray. Items focus on efficacy, sensory perceptions, device characteristics, and spray delivery; such as speed of action, amount of relief, aftertaste, smell, nose tip comfort, and ease of operation.19

The qualitative development of the EARNS-Q, which included elicitation of concepts relevant to the experience with AR and preference of nasal sprays as well as subsequent testing of the acceptability and understandability of the questionnaire content constituted by these concepts in 153 patients with AR, was conducted as the first phase of the research study and is described in a separate paper. The current analyses constituted the second phase, which included evaluating the validity and psychometric properties of the EARNS-Q experience and preference modules in patients with AR.

Methods

Experience module

In an observational study, subjects aged 18–65 years with physician-diagnosed vasomotor, seasonal, and/or perennial AR and using a prescription nasal spray as part of their allergy treatment regimen completed the EARNS-Q experience module and the Treatment Satisfaction Questionnaire with Medication (TSQM) at baseline. After 2 weeks, subjects completed the EARNS-Q experience module and the TSQM. After subjects completed the baseline assessments, they returned them via courier and, upon receipt and verification of completed documents, subjects were sent packets containing the week-2 assessments.

The 28 items of the EARNS-Q experience module assess 14 attributes with regard to their product rating and their importance of efficacy, sensory perceptions, device characteristics, and spray delivery. The TSQM is a generic instrument that contains 14 items to assess effectiveness, side effects, convenience, and global satisfaction with medication.20

Preference module

A 3-week, randomized, single-blind, crossover, multicenter pilot study (GSK Study FFR105693) was conducted with subjects ≥18 years of age who had a documented clinical history of seasonal AR with seasonal (grass) allergy symptoms during each of the previous two allergy seasons, a positive skin prick test to spring seasonal (grass) pollen in the 12 months prior to beginning the study, and adequate exposure to seasonal (grass) pollen. Subjects were given a diary card at screening (visit 1) and on study days 1 (visit 2), 8 (visit 3), and 15 (visit 4) for recording morning, evening, and daily reflective total nasal symptom scores (rTNSS), a measure of AR severity. Subjects received flunisolide (Nasarel®, Teva Pharmaceuticals, North Wales, PA) or beclomethasone (Beconase®, GlaxoSmithKline, Research Triangle Park, NC) on days 1 to 7, followed by a 7-day wash-out period, followed by treatment with the other nasal spray (flunisolide or beclomethasone) on days 15 to 21. Subjects completed the EARNS-Q experience module on days 1 and 8, the EARNS-Q preference module on day 22 (visit 5), and the TSQM on days 8 and 22. The additional 15 items of the EARNS-Q preference module evaluate preference by comparing two products based on the abovementioned 14 experience attributes as well as on overall global preference.

Questionnaire scoring

For each of the 14 attributes of the experience module, the four rating response options were coded as −3, −1, 1, and 3 (most negative to most positive); the four importance options were coded as 0, 1, 2, and 3 (“not important at all” to “extremely important”).19 The importance-weighted score was calculated by multiplying the value of the rating item by the value of the importance item. Domain scores for each of the four scales – efficacy, sensory perceptions, device characteristics, and spray delivery – were calculated by determining the mean score of all items in the domain. The total experience score was the mean score of all items in the module. Because very few differences were observed between unweighted and weighted analysis results in terms of validity and reliability, only the results obtained with the unweighted algorithm are presented.

For the items of the preference module, the five response options for each item were coded left to right on a 5-point scale in which 1 = stronger preference for product one, 3 = no preference, and 5 = stronger preference for product two.

The TSQM was scored according to its standard published scoring method.19

Statistical analyses

Statistical analyses were carried out with Statistical Analysis System (SAS) (v 9 for Windows; SAS Institute, Inc, Cary, NC) although multi-trait analysis was carried out with Multitrait Analysis Program-Revised (v 1.0 for Windows, running on top of SAS v 6 for Windows). Multi-trait analysis was used to examine item convergent and discriminant validity. It is important that items are correlated with their own scales (corrected for overlap) at a level of at least 0.40 (convergent validity) and to be more highly correlated with their own scale than with other scales in the same questionnaire (corrected for overlap) (discriminant validity). For each scale, results are reported as the percentages of these tests with passing scores.

The extent to which items fit into their hypothesized scales (internal consistency), as well as the global score, was assessed using Cronbach’s α coefficient, with consistent scales having a value of at least 0.70 as the threshold for acceptable reliability of the scale. Test–retest reliability was determined for each experience module index and scale domain with an intraclass correlation coefficient (ICC) between the first and second assessments. The ICC assesses the level of consistency between the two assessments. Test–retest reliability was estimated for all subjects as well as for only those subjects who reported no change in health in the previous 2 weeks, and the criterion for acceptable test–retest reliability was an ICC of at least 0.70. Because there was no retest visit after visit 5 in the preference study, test–retest reliability of the preference items was not assessed.

Spearman’s rank correlation coefficients were used to assess the relationship between the items of the experience module and the TSQM. Correlations of 0.30 or less were considered to represent poor relationships. In addition, subjects were divided into three groups based on their TSQM treatment satisfaction scores and the mean experience and preference module scores of the groups were compared with one-way analysis of variance (ANOVA) to see how well each module score could differentiate across different levels of satisfaction. Pearson’s correlation coefficients among the EARNS-Q preference items and domains, EARNS-Q experience item and domain change scores, change in mean diary rTNSS, and change in TSQM domains were assessed to evaluate how other similar concepts were related. Experience with Allergic Rhinitis Nasal Spray Questionnaire preference items and domains were correlated with the overall preference item at visit 5.

Results

The observational study was completed by 121 subjects. Table 1 summarizes the demographic characteristics of the populations. The mean age of the observational study population was 42.8 years, and subjects had been suffering from AR for an average of 13.9 years. Most were using one nasal spray (93%) in addition to one or more oral AR medications (94%). Fluticasone propionate (Flonase®) was the most frequently prescribed nasal spray (57%).

Table 1

Demographic characteristics

Population	Characteristic	Value
Observational study population	Mean age, y ± SD (range)	42.8 ± 1.88 (19–67)
	Female gender, n (%)	96 (79)
(Experience module only)	Mean ± SD years since diagnosis (range)	13.9 ± 12.92 (0–65)
N = 121	Severity of “moderate” or “severe”, n (%)	108 (90)
	Allergen, n (%)	84 (69)
	Grass
	Mold	74 (61)
	Dust	98 (81)
	Dander	67 (55)
	Pollen	112 (93)
	Other	13 (11)
	Nasal spray, n (%)
	One spray	111 (93)
	Two sprays or more	9 (7)
	Oral medication, n (%)
	One medication	91 (80)
	Two medications or more	23 (20)
Clinical study population	Mean age, y ± SD (range)	37.2 ± 12.82 (18.0–68.0)
	Female gender, n (%)	45 (50.6)
(Experience and preference modules)	Mean ± SD baseline morning rTNSS (range)	9.2 ± 1.65 (5.3–12.0)
N = 89	Mean ± SD baseline evening rTNSS (range)	9.3 ± 1.57 (5.5–12.0)
	Mean ± SD baseline daily rTNSS (range)	9.2 ± 1.47 (6.4–12.0)

Abbreviations: SD, standard deviation; rTNSS, reflective Total Nasal Symptom Score.

The clinical study was completed by 89 subjects; mean age was 37.2 years, and mean morning, evening, and daily rTNSS baseline scores indicated stable AR severity.

Most items in the efficacy, sensory perceptions, and spray delivery domains for both modules were correlated to their own scale at an acceptable level (Table 2). In the experience module, the “ease of carrying” item was found to be poorly correlated to the device characteristics domain (Pearson’s r = 0.38). In the preference module, the “ease of carrying” item (Pearson’s r = 0.39) and the “amount left” item (Pearson’s r = 0.37) was found to be poorly correlated to the device characteristics domain.

Table 2

Item-scale correlations of the experience module and the preference module

Domain	Items, n	Range of item-scale correlations
		Experience	Preference
Efficacy	3	0.54–0.77	0.88–0.93
Sensory perceptions	5	0.40–0.57	0.67–0.85
Device characteristics	4	0.38–0.56b	0.37–0.75c
Spray delivery	2	0.56–0.56	0.80–0.80

Notes:

Unweighted values;

Ease to carry item did not meet criterion (r ≥ 0.40);

Ease to carry and amount left items did not meet criterion (r ≥ 0.40).

All items were correlated more highly with their own scale than with other scales for all domains in the experience module (Table 2). In the preference module, all items were correlated more highly with their own scale than with other scales for the efficacy, sensory perceptions, and spray delivery domains, however, for the device characteristics domain, the “ease of carrying”, “ease of operation”, and “amount left” items did not correlate more highly with their own scale.

All items fit well in their hypothesized domains of both modules of the EARNS-Q (Table 3, internal consistency), as evidenced by a Cronbach’s α of 0.71 to 0.82 for the domain scores and 0.72 for the global score in the experience module and of 0.74 to 0.96 for the domain scores and 0.93 for the global score in the preference module. Test–retest reliability from baseline to follow-up on the EARNS-Q experience module domains yielded ICC values that ranged from 0.64 to 0.82 in the analysis of all subjects and from 0.64 to 0.82 in the analysis of only those subjects who experienced no change in health (Table 4). Therefore, for all subjects, the efficacy and sensory perceptions domains demonstrated good test–retest reliability (ICC ≥ 0.70), and for subjects who experienced no change in health, all domains except the device characteristics domain met the criterion for good test–retest reliability. The total experience score surpassed the criterion in both analyses.

Table 3

Internal consistency reliability of the multi-item domains of the experience module at baseline and the preference module at visit 5

Domain	Total items, n	Cronbach α
		Experience	Preference
Efficacy	3	0.82	0.96
Sensory perception	5	0.73	0.90
Device characteristics	4	0.71	0.74
Spray delivery	2	0.71	0.89
Total	14	0.72	0.93

Note:

Unweighted values.

Table 4

Test–retest reliability for the multi-item domain scores of the experience module

Domain	All subjects		Subjects who experienced no change in healthb
	n	ICCc	n	ICCc
Total experience	118	0.82	101	0.82
Efficacy	117	0.79	100	0.78
Sensory perceptions	115	0.81	98	0.78
Device characteristics	116	0.64	99	0.64
Spray delivery	118	0.68	101	0.71

Notes:

Unweighted values;

Excludes patients who reported “much better” or “much worse” allergies or general health compared with 2 weeks prior;

Shrout-Fleiss ICC (2,1).

Abbreviation: ICC, intraclass correlation coefficient.

A pattern of strong and consistent correlations was observed between the EARNS-Q experience module domains and the effectiveness, convenience, and global satisfaction domains of the TSQM; however, fewer and weaker correlations were observed between the EARNS-Q experience module domains and the side effect domain of the TSQM (Table 5).

Table 5

Correlation between experience domain change scores and preference domains (Pearson coefficients), and experience module domain scores and TSQM scores (Spearman coefficients)

Scale	Domain	EARNS-Q
		Total	Efficacy	Sensory perceptions	Device characteristics	Spray delivery
Preference domainc	Total	0.72	0.38	0.66	0.42	0.38
	Efficacy	0.47	0.55	0.29	0.20	0.29
	Sensory perceptions	0.75	0.25	0.79	0.43	0.29
	Device characteristics	0.63	0.20	0.59	0.45	0.32
	Spray delivery	0.47	0.24	0.38	0.32	0.39
	Global preference (item)	0.59	0.44	0.47	0.33	0.35
TSQM domaind	Effectiveness	0.62	0.67	0.30	0.52	0.43
	Side effects	0.24	0.10	0.23	0.19	0.02
	Convenience	0.57	0.39	0.36	0.56	0.37
	Global satisfaction	0.60	0.56	0.31	0.46	0.43

Notes:

Change from Visit 3 to Visit 5;

Unweighted values;

Pearson correlation coefficients of preference domains with Experience domain change scores;

Spearman correlation coefficients of TSQM domains with Experience domain scores.

Abbreviations: TSQM, Treatment Satisfaction Questionnaire with Medication; EARNS-Q, Experience with Allergic Rhinitis Nasal Spray Questionnaire.

Strong correlations were observed between the same domains of the EARNS-Q preference module and the change in those EARNS-Q experience domains; the overall (global) product preference item correlated most highly with change in total experience score (r = 0.59) (Table 5).

Assessment of the experience domains by level of satisfaction on the TSQM at baseline (low satisfaction, TSQM global score <50; moderate satisfaction, TSQM global score >50 and <70; high satisfaction, TSQM global score >70) indicated that the low satisfaction group had the lowest mean EARNS-Q scores, and the high satisfaction group had the highest mean EARNS-Q scores for all domains (Table 6).

Table 6

Ability of the experience module to discriminate across levels of satisfaction

Experience module domains	Baseline TSQM categories	Mean (SD)	F statistic	P value
Total experience	Low satisfaction (n = 42)	0.56 (0.88)		<0.0001
	Moderate satisfaction (n = 41)	0.96 (0.50)	51.56
	High satisfaction (n = 38)	1.67 (0.61)
Efficacy	Low satisfaction (n = 42)	−0.14 (1.24)	38.35	<0.0001
	Moderate satisfaction (n = 41)	0.64 (1.00)
	High satisfaction (n = 38)	1.30 (0.83)
Sensory perceptions	Low satisfaction (n = 41)	0.84 (1.13)	12.75	0.0005
	Moderate satisfaction (n = 41)	1.12 (1.00)
	High satisfaction (n = 37)	1.66 (0.84)
Device characteristics	Low satisfaction (n = 41)	0.80 (1.09)		<0.0001
	Moderate satisfaction (n = 41)	0.99 (0.73)	26.70
	High satisfaction (n = 38)	1.89 (0.89)
Spray delivery	Low satisfaction (n = 42)	0.29 (1.55)		<0.0001
	Moderate satisfaction (n = 41)	0.98 (1.37)	23.01
	High satisfaction (n = 38)	1.71 (0.98)

Note:

Unweighted values.

Abbreviations: TSQM, Treatment Satisfaction Questionnaire with Medication; SD, standard deviation.

Stronger correlations between the preference module domains and change in TSQM and rTNSS domains were observed when the domains were more conceptually related (Table 7). Overall product preference was most highly correlated with TSQM global satisfaction domain change score (r = 0.54), and the device characteristics domain was correlated with the TSQM convenience domain change score (r = 0.49). All EARNS-Q preference domains were associated with reductions in mean daily rTNSS.

Table 7

Pearson correlation coefficients of preference domains and overall preference item with change on TSQM and rTNSS

Preference domain	TSQM domain change score				Change in mean daily rTNSS
	Global satisfaction	Convenience	Effectiveness	Side effects
Total	0.45	0.45	0.44	0.36	−0.39
Efficacy	0.56	0.30	0.59	0.26	−0.53
Sensory perceptions	0.33	0.38	0.33	0.37	−0.27
Device characteristics	0.29	0.49	0.24	0.26	−0.21
Spray delivery	0.34	0.37	0.30	0.25	−0.27
Overall product Performance (item)	0.54	0.35	0.52	0.30	−0.50

Note:

Change from Visit 3 to Visit 5.

Abbreviations: TSQM, Treatment Satisfaction Questionnaire with Medication; rTNSS, reflective Total Nasal Symptom Score.

All specific product attribute preference items had substantial and expected relationships to overall product preference. The overall product preference item was most highly correlated with the three efficacy preference items (“spray works quickly”, r = 0.79; “allergy relief ”, r = 0.87; and “spray works well”, r = 0.89) (Table 8).

Table 8

Pearson correlation coefficients of EARNS-Q specific product attribute preferences with overall product preference item

EARNS-Q specific product attribute preference item		EARNS-Q overall product preference item
Item	Description
Carry	Ease to carry preference	0.49
Dose	Prescribed dose delivery preference	0.67
Nose	Tip comfort preference	0.56
Operation	Ease of operation preference	0.56
Dosing	Same amount each time preference	0.66
Smell	Smell strength preference	0.45
Onset	Spray works quickly preference	0.79
Aftertaste	Aftertaste preference	0.58
Relief	Allergy relief preference	0.87
Left	Amount medication left preference	0.38
Leakage	Leak out nose and throat preference	0.67
Burn	Nasal spray burns preference	0.68
Mist	Gentle mist preference	0.58
Work	Spray works well preference	0.89

Abbreviation: EARNS-Q, Experience with Allergic Rhinitis Nasal Spray Questionnaire.

Discussion

The majority of studies evaluating patient preference with intranasal corticosteroids have evaluated only sensory perceptions.11,12,15 The EARNS-Q is the first questionnaire to evaluate multiple domains – namely, efficacy, sensory perceptions, device characteristics, and spray delivery – involved in determining experience and preference with regard to nasal sprays in subjects with AR.

Results from these analyses have shown that the EARNS-Q preference and experience modules demonstrate excellent internal consistency reliability. Correlations among the preference and experience module domains were moderate, indicating that the preference and experience module scales measure related but distinct constructs. With regard to test–retest reliability, which was measured only for the experience module, the total experience score surpassed the criterion in both types of analyses (all subjects and only subjects reporting no change in health), indicating that the domain scores are generally reproducible and stable over a period of several weeks when no change in health or functioning has occurred. The domains of each module are reliable and valid in relation to external measures of similar constructs (TSQM). The pattern of correlations between the preference module domains and the change in the experience module domains, as well as the pattern of correlations between the preference module domains and the change in TSQM and rTNSS domains, lends additional support to its validity.

The relationship between the experience and preference modules supports the underlying conceptual model for the assessment of preference. This is the first allergic rhinitis specific tool to assess not only preference for treatment, but also characterizes patient experience and drivers of patient preferences. Although the questionnaire has not been used to evaluate differences in responses between allergic and vasomotor rhinitis, the use of the EARNS-Q in practice (experience module) or clinical trials (both modules) will allow the patient and physician to make better informed treatment decisions.

In contrast to the TSQM, a nonspecific general questionnaire used to determine patient treatment satisfaction,19 the EARNS-Q evaluates criteria specific to intranasal corticosteroids and patients with AR. The results of these studies indicate that the EARNS-Q is psychometrically sound within the AR population. Therefore, the EARNS-Q provides a comprehensive assessment for predicting which intranasal corticosteroid would have the highest compliance. This new instrument is well-suited for helping patients and physicians assess experience with and preference for nasal sprays.