Psychometric properties of the Behavioural Outcomes of Anxiety questionnaire in stroke patients with aphasia.

OBJECTIVE: To evaluate the psychometric properties of an observational, carer-completed anxiety screen for aphasic stroke patients.
DESIGN: Phase 1: A cross-sectional questionnaire design to establish psychometric properties. Phase 2: A randomized longitudinal design with treatment and control to evaluate sensitivity to change and repeatability/reliability.
SUBJECTS: Phase 1: 111 patient-carer dyads were recruited through stroke charities: patient mean age 69.7(10. 7), 6.2(5. 2) years since stroke, 76 male; carer mean age 64.7(12. 2), 27 male. Phase 2. A subsample of 50 dyads (29 completed). MEASURES: All patients completed the Tension Rating Circles and the Frenchay Aphasia Screening Test. Carers completed the Behavioural Outcomes of Anxiety questionnaire, observational versions of the Hospital Anxiety and Depression Scale (HADS-A) and the Generalised Anxiety Disorder-7, and a feedback questionnaire. INTERVENTION: Phase 2: 25 dyads were offered relaxation training and 25 acted as controls.
RESULTS: The Behavioural Outcomes of Anxiety questionnaire correlated .77 with the HADS-A and Cronbach's Alpha was .82 demonstrating validity and internal consistency. Using HADS-A cut-off > 7 as criterion the area under the curve was 0.90 and at cut-off of > 16 sensitivity (0.85) and specificity (0.85) were both good. Scores declined significantly more in a group given anxiety training ( n = 12) than in a control group ( n = 17), demonstrating sensitivity to change and construct validity. Two-week repeatability/reliability was .92. Feedback suggested the scale was acceptable.
CONCLUSIONS: The Behavioural Outcomes of Anxiety questionnaire shows promise as an anxiety screen for stroke patients with aphasia and is sensitive to change. Further analysis of dimensionality and discriminant validity is needed.

RCT Entities:   Population Interventions Outcomes

Introduction

Anxiety prevalence estimates range from 18-38%[1] and during the first 10 years after stroke the cumulative incidence is 57%.[2] Anxiety is also persistent,[3] and is associated with poor social functioning,[4] lower quality of life, depression[1,5] and poorer functional ability.[6] Assessing anxiety after stroke can be difficult because about 30% of patients have aphasia. [7] Reliable assessment of emotional state by clinical interview or by self-report questionnaires can be impossible in the presence of aphasia.[8-10] Even among those without substantial communication problems, 60% experience difficulty understanding self-report measures.[11] Currently there exists no validated anxiety screen for aphasic stroke patients.[12]

An alternative to self-report is observational rating by others.[13] The Behavioural Outcomes of Anxiety questionnaire[12,14] is such an observational tool. It was developed to screen for anxiety in those with aphasia while avoiding bias attributable to the somatic and behavioural effects of stroke. Items were designed to measure signs of anxiety that are readily observable by carers via 10 descriptors of anxiety, based on diagnostic criteria.

The Behavioural Outcomes of Anxiety questionnaire has been evaluated in stroke patients without communication difficulties,[14] but its psychometric properties and acceptability for carers of stroke patients with aphasia have yet to be determined. This was the primary aim of this study. A secondary aim was to determine the Behavioural Outcomes of Anxiety questionnaire’s sensitivity to change by considering the impact upon scores of an intervention for anxiety that appears effective in people with stroke, namely relaxation training.[15] Finally, we also aimed to undertake preliminary evaluations of an observational version of the self-report Generalised Anxiety Disorder-7[16] and the Tension Rating Circles.[17] The latter uses self-rated muscle tension to gauge anxiety. Neither of these instruments has been studied with an aphasic stroke population.

Method

Phase 1: Investigation of reliability and validity

Phase 1 used a cross-sectional design to assess psychometric properties. In Phase 2 a subsample of 50 participated in a two-group quasi-experimental design; relaxation intervention and control group. (See Figure 1 for details of recruitment).

Figure 1.

Recruitment.

The validity of the scales was evaluated through association with the Hospital Anxiety and Depression Scale-Anxiety subscale (HADS-A)[18] which has established validity in stroke.[19] Repeatability was evaluated using intra-class correlation for a repeated administration at an interval of fourteen days for a subsample, and the small two-group design assessed sensitivity to change.

Ethical approval was obtained from the Cardiff University School of Psychology Ethics Committee.

Sample size was comparable with other validation studies of stroke-specific questionnaires (N = 89[14], N = 49;[20] (N = 70)).[21] Power analysis with GPower[22] showed that a bivariate correlation of 0.29 could be detected at power = 0.90, alpha = 0.05 (one-tailed) with 98 participants. Receiver Operating Characteristic analysis required a sample of 22 to distinguish a typical Area Under the Curve of 0.8 from 0.5 (no prediction) at power = 0.80, alpha = 0.05.

Stroke patients with communication difficulties and their carers were recruited from stroke groups in south Wales. The sampling was opportunistic; the research was advertised to the groups and potential participants contacted the researcher or were proposed by staff. Inclusion criteria were: stroke between 2 months and 20 years of recruitment; communication impairment indexed by the age adjusted cut-off point on the Frenchay Aphasia Screening Test of < 25;[23] patient with a carer who spends at least three hours a week with them; patient can point to the circle that corresponds to their level of tension on the Tension Rating Circles and complete tick boxes on the demographic questionnaire (with assistance if necessary). Exclusion criteria were patient or carer under 18 years of age or not meeting the inclusion criteria.

A 12 item demographic questionnaire obtained information about each patient. It was completed by patients with assistance from a carer or researcher if necessary.

The Behavioural Outcomes of Anxiety questionnaire[12] has 10 questions (Appendix 1, supplementary material) and a preliminary validation with non-aphasic stroke patients is described in Linley-Adams et al.[14]

A carer-completed version of the Hospital Anxiety and Depression Scale-Anxiety subscale[14] was used as the standard. The HADS-A has been validated in stroke[19] and is recommended for anxiety screening.[24] It was chosen since at this time it is the only anxiety scale validated for stroke, and the carer completed version has been validated against stroke patients’ self-reported anxiety.[14] Moreover, the alternative of a psychiatric interview is not feasible with aphasic samples.[8-10]

A carer-completed version of the Generalised Anxiety Disorder-7[16] assessed generalised anxiety. It has yet to be validated in a stroke population.

The Tension Rating Circles requires individuals to point to the circle representing their degree of muscle tension.[17]

The Frenchay Aphasia Screening Test[23,25] was used to measure communication ability.

Finally there was a questionnaire to assess the acceptability and ease of completion of the Behavioural Outcomes of Anxiety questionnaire. Four items were scored on a Likert scale (1- strongly disagree to 5 - strongly agree) and there was space for comments on the experience of using the scale.

Patients and carer dyads completed all measures, including any repeated tests, in a private room in the stroke group venue or at home. Instructions for observational tests emphasised the need for independence in rating.

Phase 2: Investigation of sensitivity to change

A sub-group of 50 dyads who volunteered for re-test and relaxation training were allocated to either a relaxation condition or a waiting control group using a random sequence. The relaxation group received a relaxation CD of progressive muscular relaxation exercises[26] and a schedule to follow for two weeks. Weekly telephone contact and a ‘relaxation diary’ supported and recorded compliance. The control group had no intervention at this stage. Both groups repeated the anxiety measures after fourteen days. The control group was subsequently provided with the relaxation materials. The experimenter was not blind to group allocation.

SPSS, Version 20 (2011, IBM Corporation, Armonk, NY, USA) was used for the majority of the statistical analyses. Receiver Operating Characteristic analysis used MedCalc version 12.7.4.0 (Medcalc Software bvba, Ostend, Belgium).

Results

Sample characteristics and test scores are presented in Table 1.

Table 1.

Sample characteristics and test scores.

Sample characteristics
Main sample (n=111 dyads)			Relaxation subsample
Variable	N		Relaxation (n=12)	Control (n=17)
Male gender (Patient* : Carer*)	76: 27		8 : 3	11 : 7
Female gender (Patient* : Carer*)	34 : 80		4 : 8	6 : 9
Self-reported communication difficulty	108		11	17
Carer-reported communication difficulty	101		10	16
First stroke	65		5	12
Living with carer/someone	94		10	12
	N	Mean(SD)	Mean(SD)	Mean(SD
Patient age	110	69.7 (10.7)	68.7 (10.4)	69.4 (11.6)
Carer age	108	64.7 (12.2)	64.6 (11.0)	62.6 (14.9)
Years since stroke	107	6.2 (5.2)	6.6 (6.1)	6.4 (4.5)
FAST	111	19.0 (5.9)
TRCs	111	2.0 (1.4)
BOA	111	15.1 (6.2)	17.2 (4.4)	15.9 (7.3)
Post-test BOA	29	12.9 (7.6)	8.6 (6.7)	16.0 (6.7)
GAD-7	111	5.5 (5.4)	10.5 (5.0)	6.1 (6.7)
Post-test GAD-7	29	5.5 (6.1)	3.5 (4.5)	7.0 (6.7)
HADS-A	111	6.9 (4.4)	10.7 (3.8)	7.3 (5.9)
Post-test HADS-A	29	5.2 (5.2)	3.0 (3.7)	6.7 (6.2)

FAST = The Frenchay Aphasia Screening Test, TRCs = Tension Rating Circles, GAD-7 = Generalised Anxiety Disorder-7, HADS-A = Hospital Anxiety and Depression Scale-Anxiety subscale, BOA = Behavioural Outcomes of Anxiety questionnaire.

Four carers and one patient did not provided gender data, one carer in each relaxation condition.

The additional psychometric properties computed for the Tension Rating Circles, Generalised Anxiety Disorder-7 and Behavioural Outcomes of Anxiety questionnaire are presented in Table 2. These include validity against the Hospital Anxiety and Depression Scale-Anxiety subscale, Cronbach’s Alpha (internal consistency), test-retest reliability/repeatability, area under the Receiver Operating Characteristic curve and sensitive, specificity and positive and negative predictive values. All items of the Behavioural Outcomes of Anxiety questionnaire were correlated with the corrected scale total, corrected by removal of the item being correlated, scores ranged from .39 to .66. Cronbach’s alpha was high and a principal component analysis demonstrated that a single component of Eigenvalue 4.10 accounted for 41.0% of the variance in the 10 items.

Table 2.

Psychometric properties of the scales.

Test	Validity Against HADS (n = 111)	Cronbach’s Alpha (n- = 111)	Test-Retest Reliability (Intra-class-correlation) (n = 17)	ROC Analysis† Area Under Curve (95% CI)	Cut-off	Sensitivity (95% CI)	Specificity (95% CI)	Positive predictive value	Negative predictive value
BOA	.77	0.82	.91	0.90 (0.83–0.95)**	16/17	0.85 (0.71–0.94)	0.85 (0.73–0.92)	0.38	0.98
GAD-7	.82	0.87	.87	0.94 (0.87–0.97)**	4/5	0.91 (0.79–0.98)	0.83 (0.72–0.91)	0.37	0.99
TRCs	.31			0.62 (0.52–0.71)*	1/2	0.76 (0.61–0.87)	0.41 (0.29–0.54)	0.13	0.94

= P < 0.001; *=P < 0.05.

Using the HADS-A at the cut-off of 7/8 proposed by Zigmond and Snaith[18] and recommended for community samples.[14]

TRCs = Tension Rating Circles, GAD-7 = Generalised Anxiety Disorder-7, HADS-A = Hospital Anxiety and Depression Scale-Anxiety subscale, BOA = Behavioural Outcomes of Anxiety Questionnaire, ROC= Receiver Operating Characteristic.

In addition, validity of the Behavioural Outcomes of Anxiety questionnaire against the Hospital Anxiety and Depression Scale-Anxiety subscale and its repeatability are illustrated in scatterplots (Figures 2 and 3). Finally, the Receiver Operating Characteristic curve of the Behavioural Outcomes of Anxiety questionnaire against the Hospital Anxiety and Depression Scale-Anxiety subscale is presented in Figure 4.

Figure 2.

Behaviour Outcomes of Anxiety questionnaire (BOA) score plotted against HADS-A Scores (N = 111, r = .77).

Figure 3.

Behaviour Outcomes of Anxiety questionnaire (BOA) scores at first administration and at 14 day re-test for the Control Group (n = 17, r = .91).

Figure 4.

Receiver Operating Characteristic Curve for the Behavioural Outcomes of Anxiety questionnaire against the HADS-A at cut-off 7/8.

The construct validity of the Behavioural Outcomes of Anxiety questionnaire was further investigated by examining inter-correlations of all the anxiety measures. All the anxiety measures were all inter-correlated. For all N = 111, p < .01, two-tailed:

Behavioural Outcomes of Anxiety questionnaire correlated with:

HADS-A (r = .77);

Generalised Anxiety Disorder-7 (r =. 71);

Tension Rating Circles (r = .31).

HADS-A correlated with:

Generalised Anxiety Disorder-7 (r = .82);

Tension Rating Circles (r = .31).

Generalised Anxiety Disorder-7 correlated with:

Tension Rating Circles (r = .30)

Age correlated negatively and significantly with Behavioural Outcomes of Anxiety questionnaire, Generalised Anxiety Disorder-7 and HADS-A scores (r = -.28 to -.22, n = 110, p = 0.012).

The questionnaire to assess the experience of using the Behavioural Outcomes of Anxiety questionnaire was completed by 109 carers; 104 agreed or strongly agreed with the statement ‘I felt confident completing the Behavioural Outcomes of Anxiety questionnaire’; 106 agreed that the questions made sense to them and 103 agreed or strongly agreed that the ‘questionnaire was easy to complete’. Only 3 said the Behavioural Outcomes of Anxiety questionnaire was difficult to complete and 103 disagreed or strongly disagreed with this statement.

The open question was answered by 27 carers and the majority of responses were positive.

‘The questionnaire was clearly explained and easy to complete’ (n = 6); ‘No problem at all’ (n = 6); ‘It was easy to fill in the questionnaire as each question was clear and concise’ (n = 2); ‘Not difficult, the questions were quite clear and straight forward’ (n = 4); ‘It’s hard to figure out what my [spouse] is thinking sometimes’ (n = 4).

Investigation of sensitivity to change

Of the 25 dyads randomly assigned to the relaxation group 13 did not complete the intervention and re-test, whereas only 7 of the control group did not complete the re-test. The 25 patients selected for relaxation training did not differ significantly from controls on pre-intervention measures. But following attrition, the relaxation group (n = 12) had higher pre-intervention anxiety across all three measures (V = .166, F(6, 214) = 3.23, p = .005) than controls (n = 17). Those that failed to complete relaxation and dropped out had lower pre-intervention scores on all three anxiety measures than those who completed relaxation (p < 0.02, two tailed, for all comparisons).

Those completing the relaxation condition demonstrated a significantly greater multivariate reduction in anxiety across all three anxiety measures than those completing the control condition; Multivariate Group x Time interaction, (V = 0.66, F(3,25) = 16.31, p < .000). Follow-up tests of interactions showed significant reductions for all three measures: Behavioural Outcomes of Anxiety questionnaire (F(1,27) = 23.40, p < .000); HADS-A (F(1,27) = 47.67, p < .000) and Generalised Anxiety Disorder-7 (F(1, 27) = 45.21, n < .000).

Discussion

The Behavioural Outcomes of Anxiety questionnaire showed high construct validity in terms of its correlation with the HADS-A, good internal consistency as indexed by Cronbach’s alpha and excellent repeatability/test re-test reliability over a two week period. The high Cronbach’s alpha, together with the item-total correlations and the large principal component accounting for over 40% of variance, are suggestive of a unidimensional scale though this requires verification with a larger sample. The scale also demonstrated sensitivity to change in response to the relaxation intervention. The 0.90 Area Under the Curve against the HADS-A fell in the ‘high’ range[27] and at a cut-off score of 16/17 diagnosis exceeded criteria for sensitivity and specificity.[24] This cut-off score was higher than 13/14 recommended by Linley-Adams et al.[14] for non-aphasic stroke patients and gave better sensitivity and specificity. The Behavioural Outcomes of Anxiety questionnaire was also comparable with, or superior to, alternative stroke-validated mood screens in terms of Cronbach’s alpha,[10] Area Under the Curve,[9,10] sensitivity[9,10] and specificity.[9,10,28] It exhibited high negative predictive value (0.98), but lower positive predictive value (0.38) which indicates further assessment of those above the cut-off. Reliable changes in scores in response to the relaxation intervention demonstrated the sensitivity to change of the scale, as well as further evidence of its construct validity. Most carers were positive about completing the scale, few mentioned drawbacks, attesting to its acceptability. In summary, the Behavioural Outcomes of Anxiety questionnaire has promise as a valid, reliable and acceptable brief screen for anxiety in aphasic stroke patients that can help identify those who would benefit from one of the available anxiety treatments such as behaviour therapy which is effective in the presence of aphasia,[29] or relaxation.[15,17] The evidence for its sensitivity to change also suggests that it could be an effective outcome measure for aphasic patients.

The observational Generalised Anxiety Disorder-7 correlated strongly with the Behavioural Outcomes of Anxiety questionnaire and showed comparable properties (Area Under the Curve=0.94–also in the ‘high’ range) and at cut-off of 4/5 sensitivity and specificity also exceeded the recommended standards.[24] It now requires validation as a self-report measure with non-aphasic stroke samples. The self-report Tension Rating Circles correlated weakly with the other anxiety scales. The Area Under the Curve (0.62) was in the ‘low’ range and a cut-off could not be found that gave acceptable specificity and sensitivity. Aphasic stroke patients may find self-reporting tension difficult, even with a simplified response. Therefore the Tension Rating Circles cannot be recommended as an anxiety screen.

The relaxation training intervention appeared to be effective in ameliorating anxiety. Larger scale studies with diagnosed anxious patients are required to provide confirmation of this. Attrition from the intervention groups is explained by less anxious patients dropping out, probably (and understandably) due to reduced motivation for the treatment. However, the efficacy of the intervention cannot be attributed to selective attrition; those who did not complete the relaxation intervention had lower pre-intervention anxiety scores.

At the 7/8 HADS-A cut-off anxiety prevalence was 41.4%, higher than the 20-35% typically found in general, non-aphasic stroke samples.[1,2] This merits investigation through studies of matched aphasic and non-aphasic samples. All the anxiety scales correlated negatively with age of patients consistent with research suggesting increased vulnerability of younger stroke patients to anxiety.[30]

The study had some strengths and weaknesses. Generalisation was improved by a heterogeneous sample in terms of time since stroke, age, number of strokes and type of stroke. It also included patients with moderate to severe communication difficulties; the average Frenchay Aphasia Screening Test score (19/30) was markedly below the cut-offs for aphasia (25 out of 30). However, the sample was opportunistic and depended on consent, both of which could introduce bias relative to the ‘typical’ stroke patient. Further it was not possible to collect reliable data on the type or location of stroke, despite inclusion of a question about this, since most patients checked, ‘don’t know’. A systematic stroke register with a minimum data set would greatly facilitate collection of such medical data. Recent strokes were not included, meaning the performance and acceptability of the Behavioural Outcomes of Anxiety questionnaire in this group is yet to be determined. It is also a concern that observational measures of emotional difficulties may be influenced by carer’s mood;[10] A distressed carer may complete screening in a manner that reflects their mood[21] and carers may overestimate patients’ distress.[10,14] The possibility of estimation bias has been neglected in the validation of observational assessments and merits further research. The sample obtained to examine test-re-test was also small, meaning replication with a larger group is desirable. The Behavioural Outcomes of Anxiety questionnaire has yet to be assessed in samples of predominantly professional care staff (only four were in this sample). This is an important future project. Attrition of non-anxious patients from the relaxation groups indicates a need for further relaxation studies with only highly anxious patients. The present study did not measure depression. This was to reduce test burden and fatigue. Further studies could seek to establish the discriminant validity of the Behavioural Outcomes of Anxiety questionnaire against depression.

The Behavioural Outcomes of Anxiety questionnaire is brief, practicable and acceptable for use with the aphasic stroke population.

It has good concordance with established anxiety screens which suggests validity, and there is preliminary evidence that scores are stable across repeated administrations, supporting repeatability/reliability.

Its sensitivity and specificity are both within the acceptable range for anxiety screening.

Score changes following an intervention to reduce anxiety, suggest adequate sensitivity of the Behavioural Outcomes of Anxiety questionnaire