The AWESCORE, a patient-reported outcome measure: development, feasibility, reliability, validity and responsiveness for adults with cystic fibrosis.

BACKGROUND: Quality of life has improved dramatically over the past two decades in people with cystic fibrosis (CF). Quantification has been enabled by patient-reported outcome measures (PROMs); however, many are lengthy and can be challenging to use in routine clinical practice. We propose a short-form PROM that correlates well with established quality-of-life measures.
METHODS: We evaluated the utility of a 10-item score (AWESCORE) by measuring reliability, validity and responsiveness in adults with CF. The questions were developed by thematic analysis of survey questions to patients in a single adult CF centre. Each question was scored using a numerical rating scale 0 to 10. Total scores ranged from 0 to 100. Test-retest reliability was assessed over 24 h. To determine validity, comparisons were sought between stable subjects and those in pulmonary exacerbation, and between AWESCORE and Cystic Fibrosis Questionnaire - Revised (CFQ-R). Responsiveness to pulmonary exacerbation in individual subjects was evaluated.
RESULTS: Five domains, each with two questions, were identified for respiratory, physical, nutritional, psychological and general health. A total of 246 consecutive adults attending the outpatient clinic completed the AWESCORE. Scores were higher during clinical stability compared to pulmonary exacerbation (mean± sd): 73±11 versus 48±11 (p<0.001). Each domain scored worse during an acute exacerbation (p<0.001). No differences in reliability were observed in scores on retesting using Bland-Altman comparison. The CFQ-R scores (mean±sd: 813±125) and AWESCORE (81±13) were moderately correlated (Pearson's r=0.649; p=0.002).
CONCLUSIONS: The AWESCORE is valid, reliable and responsive to altered health status in CF.

Introduction

People with cystic fibrosis (CF) live longer and relatively normal lives studying, working, travelling, forming close relationships and becoming parents and grandparents. This improvement in survival and quality of life is as a result of state-of-the-art assessment and treatment by well-trained and dedicated specialised members of the CF multidisciplinary team [1]. This healthcare occurs at outpatient clinic visits, via remote contact using Telehealth or during admissions to hospital. The focus today, to enable as long and normal life as possible, is empowerment of self-management, preservation of wellness, prevention of problems, aggressive treatment when necessary and adherence to prescribed therapies [1]. Patient-reported symptoms play a critical role in the clinical assessment of a pulmonary exacerbation which currently incorporates a constellation of patient symptomatology, laboratory data and physical findings [2].

The Cystic Fibrosis Questionnaire – Revised (CFQ-R) is currently the gold standard quality-of-life (QoL) questionnaire for adults with CF. However, it has many questions, requires computerised evaluation, clinical interpretation and purchase from the author, hence limiting its use in regular clinical practice [3]. Most QoL measures in healthcare focus on downstream outcomes such as survival or care processes. Recently there has been more of a focus on symptoms, functional outcomes and quality of life in different fields of medicine. Patient-reported outcome measures (PROMs) have gained status in the delivery of healthcare both for patients and clinicians and are now used to engage in shared decision-making regarding therapy options [4]. Their use has not only proved to be feasible and good for clinical care but also to enhance physician satisfaction and prevent burnout [4, 5].

Use of PROMs may improve patient–provider relationships, as patients may feel they are being heard in all areas of life that may be causing troublesome or embarrassing symptoms and distress [4, 5]. In order to improve delivery of care, the need for a quick, easy, appealing and accessible PROM was identified to quantify wellness in adults with CF in all healthcare settings.

The aims of our study were to develop a brief purpose-designed wellness questionnaire for use in clinical practice, and to evaluate feasibility, reliability, concurrent validity and responsiveness in adults with CF.

Methods

Study design

All patients with a diagnosis of CF attending the Adult Cystic Fibrosis Unit at The Alfred Hospital were eligible for inclusion. The study was composed of five parts: questionnaire development, feasibility, reliability, concurrent validity and responsiveness. Recruitment occurred prospectively between October 2013 and February 2018. There were no exclusion criteria. Approval was provided by the Alfred Human Research Ethics Committee.

Questionnaire development

The items used to create the AWESCORE were identified through multidisciplinary focus groups consisting of health professionals treating patients with CF including medical, nursing, physiotherapy, nutrition and psychology, together with an online patient advisory group (ten patients ranging in age, lung function, educational and vocational background). They met on six occasions to identify the main health domains and specific items relevant to CF. Items were scored on visual numerical rating scales, similar to those used widely to quantify pain and shortness of breath in CF [6]. The tool was piloted in ten adults with CF outside the online advisory group with subsequent adjustments for clarity.

Feasibility

Consecutive patients attending an outpatient clinic were invited to self-complete the AWESCORE. Participants were determined to be clinically stable, as assessed by stability of lung function, or in acute exacerbation, as assessed by a respiratory physician due to a significant drop in lung function and/or increased symptoms requiring oral, inhaled or intravenous antibiotics. Time to complete the questionnaire and completion rate were recorded.

Validity

Known-groups validity was assessed by comparing scores from stable participants to those experiencing an exacerbation in the group of participants from the feasibility component.

Concurrent validity was tested by comparing the AWESCORE to the CFQ-R, a well-established health-related QoL measure for people with CF. The CFQ-R consists of 50 questions and takes ∼15 min to complete [3]. Participants completed the AWESCORE and CFQ-R once in the outpatient clinic during clinical trial stability.

Reliability

In the same group of patients described in the previous paragraph test–retest reliability of the AWESCORE was tested 1 month apart pre-randomisation to a clinical trial.

In a separate group of patients test–retest reliability was assessed by participants completing the AWESCORE twice within 24 h in two settings. Clinically stable participants completed the AWESCORE twice on the same day: upon arrival at the outpatient clinic and again prior to leaving the clinic. Inpatients admitted for acute exacerbation completed the AWESCORE twice: upon admission and again within 24 h.

Responsiveness

Responsiveness was established by examining whether the AWESCORE was able to detect a clinical change in participants who completed it during both clinical stability and inpatient admission for acute exacerbation.

Statistical analysis

Data distribution was evaluated and descriptive statistics selected accordingly. Alpha was set at 0.05.

Known-groups validity was examined using the Mann–Whitney U-test.

Concurrent validity was assessed using Pearson's correlation coefficient to compare AWESCORE and CFQ-R scores.

Reliability was assessed using intraclass correlation coefficients (ICC [1, 2]) with 95% confidence intervals to compare AWESCORES (total and each of the five domains) by participants on two occasions [7]. An ICC is commonly considered poor (<0.40), fair (0.40 to 0.59), good (0.60 to 0.74) or excellent (0.75 to 1.00) [8].

Bland–Altman plots were also used to determine if there were any systematic differences across the range of values between the two time points, with limits of agreement derived from the mean difference ±1.96 of the standard deviation of the mean difference [9].

Responsiveness was assessed using effect size, standard error of measurement and minimal detectable change. The effect size quantifies the difference between two means on a unit-less scale. It is calculated as ((μ1−μ2)/σ1) where μ1 is the mean baseline score, μ2 is the mean exacerbation score and σ1 is the sd of baseline score [10]. It was interpreted using guidelines from Cohen (1992) [11] where an effect size of 0.2 is considered small, 0.5 moderate and 0.8 large. A moderate effect size is considered a clinically important effect. An effect size >1 indicates that the difference between the two means is larger than 1 sd; an effect size >2 indicates that the difference is larger than 2 sds. The standard error of measurement represents the amount of variability that can be attributed to measurement error and was calculated as (σ1×√(1−r)) where r is the ICC which was obtained from previous analyses. The minimal detectable change at the 95% confidence level (MDC95) measures the minimum amount of change in a person's score that ensures the change is not a result of measurement error, with 95% confidence. The MDC95 is calculated as (1.96×sem×√2) [12, 13].

Data analyses were undertaken using SPSS version 25 (SPSS, Chicago, IL, USA).

Results

Five health domains were identified: respiratory, physical, nutrition, psychology and general health. Two questions were developed for each domain: the respiratory domain included cough and sputum; physical domain included energy and exercise; nutrition domain comprised appetite and targeted weight; psychology domain utilised anxiety and mood; and general health domain incorporated sleep and the perception of overall health, resulting in a single-page 10-item questionnaire (supplementary material). Each question was constructed using a numerical rating scale with anchors at zero (denoting the least possible sense of wellness) and 10 (the greatest possible sense of wellness). Participants were asked to read each question on the paper-based AWESCORE questionnaire together with the anchor descriptors then to circle the appropriate number corresponding to their current perception of wellness on each of the 10 items.

Clinicians entered the score in the box at the end of each numerical rating scale and summed for the total score. Each numerical rating scale question had a maximum of 10 points, with higher scores reflecting a greater sense of wellness. The highest possible score was 100 representing perfect perceived wellness. Each completed AWESCORE (with 10 circled scores) provided a visual image of wellness at a glance highlighting low scoring symptoms or clinical problems that needed to be addressed.

A total of 246 consecutive adults who attended the outpatient clinic completed the AWESCORE. Of these, 183 participants were clinically stable, and 63 participants had an acute exacerbation (participant characteristics in table 1).

TABLE 1

Participant characteristics for each study component

Clinical status	Combined	Feasibility and known-groups validity #		Concurrent validity ¶ and reliability +	Reliability §			Responsiveness ƒ
		Stable	Exacerbation	Stable	Combined	Stable	Exacerbation	Stable	Exacerbation
n	246	183	63	20	40	27	13	60	60
Male sex n	129	100	29	12	19	13	6	28	28
Age years (mean±sd)		31±10	33±11	33±10	33±10	32±9	33±10	33±10	-
FEV1 % predicted (mean±sd)		64±23	54±20	54±19	54±19	57±22	53±18	55±17	42±16

#: participants in outpatient clinic, completed AWESCORE once (clinically stable OR exacerbation). ¶: participants in outpatient clinic, completed AWESCORE and CFQ-R once (clinically stable). +: participants in outpatient clinic, completed AWESCORE twice 1 month apart (clinically stable). §: participants in outpatient clinic, completed AWESCORE twice at one visit (clinically stable) OR inpatient admission, completed AWESCORE twice within 24 h (exacerbation). ƒ: participants in outpatient clinic, completed AWESCORE once (clinically stable) and inpatient admission, completed AWESCORE once (exacerbation).

The AWESCORE took on average 1 min to complete. No patients declined to complete the questionnaire. There were no incomplete questionnaires.

Known-groups validity

Total AWESCORES were significantly higher (mean difference 25, 95% CI 22–28) for the 183 clinically stable participants (mean±sd: 73±11) compared to the participants with an exacerbation (48±11). All domain AWESCORES were also significantly higher in the clinically stable participants (table 2).

TABLE 2

Known-groups validity: domain AWESCORES for clinically stable participants and participants with an acute exacerbation

Domain AWESCORES	Clinical stability #	Acute exacerbation ¶	Mann–Whitney U-test
Cough	7 (6–8)	4 (3–4)	<0.001
Sputum	7 (6–8)	4 (3–5)	<0.001
Energy	7 (6–8)	4 (3–5)	<0.001
Exercise	7 (5–8)	4 (3–6)	<0.001
Appetite	8 (7–10)	5 (4–7)	<0.001
Weight	8 (7–9)	5 (4–8)	<0.001
Mood	8 (7–9)	5 (4–7)	<0.001
Anxiety	8 (6–9)	6 (4–8)	<0.001
Sleep	7 (6–8)	4 (3–6)	<0.001
General health	7 (6–8)	5 (3–6)	<0.001

Data are median (IQR), p<0.001. #: n=183. ¶: n=63.

Concurrent validity

A total of 20 clinically stable participants completed the AWESCORE and CFQ-R (participant characteristics in table 1). The CFQ-R scores (mean±sd: 813±125) and AWESCORE (81±13) were moderately correlated (Pearson's r=0.649; p=0.002).

The same 20 clinically stable participants completed the AWESCORE on two separate occasions, 1 month apart, demonstrating reliability over time when patients remained in a stable baseline state (figure 1a).

FIGURE 1

a) Reliability: Bland–Altman plot for total AWESCORES (n=20, 1 month apart, clinical stability). Solid line represents mean change in AWESCORE; dashed line represents the limits of agreement (1.96×sd of the mean change in AWESCORE). b) Reliability: Bland–Altman plot for total AWESCORES (n=40). Solid line represents mean change in AWESCORE; dashed line represents the limits of agreement (1.96×sd of the mean change in AWESCORE).

A total of 40 clinically stable participants completed the AWESCORE twice within 24 h. Those that were clinically stable completed them at the beginning and end of an outpatient clinic visit (n=27), while those in exacerbation completed the AWESCORE twice within 24 h after admission to the hospital ward (n=13). Patient characteristics for this group are summarised in table 1.

Total AWESCORES were not significantly different (mean difference −0.2, 95% CI −0.971–0.571) between time A (mean±sd: 60±16) and time B (mean±sd: 60±16). The ICC for total AWESCORE was 0.989 (95% CI 0.979–0.994) with a mean difference between scores of −0.200 (CI −0.971–0.571) and limits of agreement −5.019–4.619. Results of the individual domains for time A: first AWESCORE, and time B: second AWESCORE are presented in table 3.

TABLE 3

Reliability: AWESCORES at times A and B (n=40)

AWESCORES	Time A	Time B	Wilcoxon signed rank test		ICC (95% CI)
domains			Z	p-value
Cough	5 (4–7)	5 (4–7)	−0.202	0.84	0.910 (0.836–0.951)
Sputum	6 (4–7)	5 (4–7)	−0.922	0.36	0.900 (0.820–0.946)
Energy	5 (3–8)	5 (4–7)	−0.347	0.73	0.937 (0.885–0.966)
Exercise	6 (4–8)	5 (4–8)	−0.447	0.65	0.959 (0.924–0.978)
Appetite	7 (5–9)	7 (4–9)	−1.495	0.14	0.937 (0.884–0.966)
Weight	7 (4–8)	7 (4–9)	−0.229	0.82	0.972 (0.947–0.985)
Mood	7 (5–8)	7 (5–8)	−0.865	0.39	0.874 (0.775–0.931)
Anxiety	7 (5–9)	7 (4–9)	−0.420	0.67	0.906 (0.830–0.949)
Sleep	7 (4–8)	7 (4–8)	−0.193	0.85	0.950 (0.907–0.973)
General health	Mean±sd: 6±2	Mean±sd: 6±2	Mean difference# −0.225 (95% CI −0.409 to −0.041)		0.956 (0.918–0.976)

Data are median (IQR) except where indicated. ICC: intraclass coefficient; sd: standard deviation. #: t-test as data normally distributed; p<0.05.

Figure 1b shows the Bland–Altman plot for total AWESCORES (n=40).

When the data for the stable participants (n=27) and those in acute exacerbation (n=13) were analysed separately, the reliability of the AWESCORE remained similar to the total group (n=40) (supplementary tables S1 and S2).

Inspection of the Bland–Altman plot showed that there were no systematic differences across the range of AWESCORES, with one score falling outside the limits of agreement in clinically stable participants (supplementary figure S1) and no scores falling outside the 95% limits of agreement in patients in acute exacerbation (supplementary figure S2).

A total of 60 participants completed the AWESCORE during clinical stability and during an exacerbation (participant characteristics in table 1).

A significant reduction in AWESCORE total (mean difference −30, 95% CI −32 to −25) was seen between clinical stability (mean±sd: 76±10) and exacerbation (47±13). Significant reductions were observed in all domains (all p<0.001; table 4).

TABLE 4

Responsiveness: domain AWESCORES for participants (n=60) in clinical stability and during an acute exacerbation

Domain AWESCORES	Clinical stability	Acute exacerbation	Wilcoxon signed rank test
			Z	p-value
Cough	7 (6–8)	4 (3–5)	−6.526	<0.001
Sputum	7 (6–8)	4 (3–5)	−6.454	<0.001
Energy	8 (6–9)	4 (3–5)	−6.499	<0.001
Exercise	7 (6–9)	3 (2–5)	−6.609	<0.001
Appetite	8 (8–9)	5 (3–7)	−6.073	<0.001
Weight	8 (7–9)	6 (4–8)	−5.616	<0.001
Mood	8 (8–9)	5 (4–7)	−6.203	<0.001
Anxiety	9 (7–10)	7 (5–8)	−4.735	<0.001
Sleep	8 (6–8)	4 (3–6)	−5.766	<0.001
General health	7 (7–8)	5 (3–6)	−6.481	<0.001

Data are median (IQR); p<0.05.

An effect size of 2.9 was obtained, representing a very large effect. The standard error of measurement was 5.1 points. The MDC95 was 14.1 points, which suggests that we can be 95% confident that a minimum change of 14 points on the AWESCORE corresponds to a noticeable change in wellbeing.

Discussion

The purpose of this study was to develop a brief, efficient and acceptable patient-reported outcome measure to quantify wellness in adults with CF. In a series of different studies we determined that the AWESCORE is valid, reliable and responsive to changed health status. Completion of the AWESCORE by a large proportion of adults in our CF service was feasible, time efficient and appealing to undertake in the outpatient and inpatient settings. It was shown to be valid when assessing known groups. Its concurrent validity was demonstrated when compared with the gold standard CFQ-R. Reliability was established over two time frames: 4 weeks apart and within a 24-h period. Responsiveness to changed health status was demonstrated when the AWESCORE was completed in clinically stable patients and then again at the beginning of a hospital admission for an acute pulmonary exacerbation.

The AWESCORE was developed by patient and multidisciplinary team focus groups, piloted and modified for clarity. It was found to be feasible and acceptable by a large proportion of adults in the CF service. It was completed by each patient without hesitation in 1 min or less. It is noteworthy that the numerical rating scale used is a “discriminate” or “partition” scale, which is useful for detecting change within people but not for comparisons between people. Some patients reported that completion, review and discussion of their AWESCORE values during the clinic visit indicated that the team was listening to them and encouraging them to participate actively in their healthcare. There was no reluctance to circle low scores, and in a number of instances this facilitated patient consent to referral to the team psychologist in patients who were previously reluctant to receive psychological counselling.

Recently, Rotenstein et al. [4] reported the usefulness of PROMs to facilitate diagnosis of embarrassing and previously unidentified psychosocial problems that patients had previously not discussed but were happy to disclose in a questionnaire, which resulted in improved healthcare and outcomes. Our patients commented on how appealing the focus on wellness was (even when unwell). Some commented that the condition of CF had changed markedly over the past 3 decades and that many adults are living into middle and old age with fewer symptoms, better lung function and more normal lives. They commented that older QoL questionnaires tended to focus on symptoms, morbidity and disability, which they found less appealing to complete.

The AWESCORE was found to be valid when comparing the known groups of clinically stable patients versus those in pulmonary exacerbation. When compared with an established instrument, the CFQ-R, the AWESCORE was found to be a valid QoL measure. The short duration required for completion of the AWESCORE (<1 min compared with 15 min for the CFQ-R) is a key advantage making it feasible to incorporate into the workflow of clinic visits, use in research and in conjunction with other QoL or functional measures (either CF specific or general) enabling comparison across diseases. Health professionals found it easy to administer and score and did not require an online scoring system, licence or payment. Its use in research was feasible when a snapshot of wellness was required; however, the CFQ-R provided more detailed QoL measures. In two previously published studies evaluating the effectiveness of ivacaftor on wellness, quality of life and lung function in adults with CF the AWESCORE and the CFQ-R were scored at six time points in a crossover study comparing ivacaftor with placebo in 20 patients with CF [14, 15].

The reliability of the tool was tested in three settings: research prior to randomisation to a clinical trial, outpatient clinics and inpatient admissions. Testing was done in such a way that total scores could not be calculated, memorised and then artificially duplicated by patients during the second scoring period. In the research setting the AWESCORE was found to be reliable when measured twice 4 weeks apart by clinically stable adults with CF. In the outpatient clinic setting the AWESCORE was found to be reliable when measured at the beginning and end of the clinic visit. The tool was also found to be reliable when measured by patients at the beginning of their hospital admission for a pulmonary exacerbation and again within 24 h.

The responsiveness of the AWESCORE to change was demonstrated when a large group of patients completed the AWESCORE in the outpatient clinic when well with stable lung function and again when they were unwell and admitted to hospital for intravenous antibiotics. Significant differences were shown between the total scores and each of the 10 individual domain scores during the two different states of wellness. All values moved in the expected direction with lower scores during exacerbations and higher scores during clinical stability indicating that all domains of wellness are negatively impacted by pulmonary exacerbations.

The one-page 10-item AWESCORE is only capable of producing a snapshot of each domain. Currently, this tool is completed by participants on paper. Future work will facilitate online access for use during Telehealth consultations and for incorporation into electronic medical records using patient portals. Use during Telehealth consults may provide a time-efficient measure of each individual's sense of wellness and may highlight issues that require triaging to members of the multidisciplinary team. The ease of completion of the AWESCORE and its potential for use online may make it a suitable QoL measure for Data Registries. If completion of the AWESCORE is introduced into clinical practice at the beginning of the outpatient clinic visit and is available to each health professional at the start of the consultation, it may result in the quick visualisation of the wellness scores and avoid repetition of commonly asked questions which are time consuming and which patients may find tedious.

In the research field the AWESCORE is being used pre- and post-introduction of gene modulators to efficiently monitor the effects on the five domains of wellness. It was compared with the CFQ-R with the introduction of ivacaftor and found to be valid [14]. In the future it will be used to predict how much change from baseline is an indication of an imminent exacerbation and which of the 10 questions is the strongest predictor of an exacerbation. Construct validity testing will be undertaken in a future trial to establish the correlations between the subjective measures of the AWESCORE and objective measures of weight and forced expiratory volume in 1 s (FEV1).

In conclusion, our group has developed the new, efficient, valid, reliable and responsive AWESCORE to be used as a PROM to determine wellness in the outpatient and inpatient settings in adults with CF. Limitations of this tool primarily relate to its brevity. There are a number of other items the study group would have liked to have included such as musculoskeletal problems, pain and social isolation. As the focus groups resolved to only have 10 items with a total possible score of 100 (perfect health), prioritisation was required and these important items were excluded. With ever-growing CF outpatient clinic numbers, completion of the AWESCORE may enhance workflow efficiency by allowing members of the team to review the results and focus on further assessment and treatment of discipline-specific problems. The AWESCORE when used in the inpatient setting during acute exacerbations may assist in determining the length of treatment using intravenous, inhaled or oral antibiotics. Regular use of this PROM has the potential for greater patient satisfaction and improvements in long-term outcomes.

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary material 00120-2021.supplement