Psychometric validation of the needs assessment tool: progressive disease in interstitial lung disease.

The inter-rater/test-retest reliability and construct validity of a palliative care needs assessment tool in interstitial lung disease (NAT:PD-ILD) were tested using NAT:PD-ILD-guided video-recorded consultations, and NAT:PD-ILD-guided consultations, and patient and carer-report outcomes (St George's Respiratory Questionnaire (SGRQ)-ILD, Carer Strain Index (CSI)/Carer Support Needs Assessment Tool (CSNAT)). 11/16 items reached at least fair inter-rater agreement; 5 items reached at least moderate test-retest agreement. 4/6 patient constructs demonstrated agreement with SGRQ-I scores (Kendall's tau-b, 0.24-20.36; P<0.05). 4/7 carer constructs agreed with the CSI/CSNAT items (kappa, 0.23-20.53). The NAT:PD-ILD is reliable and valid. Clinical effectiveness and implementation are to be evaluated. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Introduction

People with interstitial lung disease (ILD) are symptomatic,1 and have limited disease-modifying treatment options, poor prognosis and poor quality of life.2 Identification and management of patients’ and carers’ palliative care needs are rare despite policy directives promoting palliative care,3 and availability of palliative interventions.4

The validated needs assessment tool in cancer (NAT:PD-C) helps clinicians identify and address palliative needs in daily practice.5 We adapted, validated and tested the reliability of the NAT:PD-C for patients with ILD (NAT:PD-ILD) and explored implementation implications in practice. This single page guide prompts clinicians to assess holistic needs (priority prompts for specialised palliative care input, patients’ well-being, informal carers’ needs and information needs), triage ongoing care (‘directly managed’, ‘refer to other team member’, ‘refer to specialist palliative care’) and also acts as a referral form.

Methods

Summary design

The initial adaptation, face and content validation6 and implementation work7 are reported elsewhere. This study tested the psychometric properties of (1) inter-rater and test–retest reliability, and (2) construct validity.

Clinicians (doctors, nurses, physiotherapists), patients with ILD and their family carers were recruited from four ILD tertiary referral clinics.

Inter-rater and test–retest reliability

Video recordings were made of 10 patient-clinician consultations (range of disease severity and carer present/absent, with clinicians using the NAT:PD-ILD to guide assessment). Clinicians were trained to use the NAT:PD-ILD and rated at least one video consultation (group viewing or individual viewing via secure online service). Clinicians were asked to rerate the same video at least 2 weeks later. Weighted Fleiss’ kappa with quadratic weights was calculated for the ratings on 10 videos (kappa <0=poor agreement, 0–0.20=slight, 0.21–0.40=fair, 0.41–0.60=moderate, 0.61–0.80=substantial, 0.81–1=almost perfect agreement).8 Data simulations indicated that we required 60 paired assessment ratings to detect at least ‘substantial’ inter-rater agreement (kappa >0.6) for the item 4 (‘Is the patient experiencing unresolved physical symptoms?’) with 80% power.

Construct validation

Trained clinicians conducted a NAT:PD-ILD-guided clinic consultation. Patients completed the St George’s Respiratory Questionnaire (SGRQ-I) and carers completed the Carer Strain Index (CSI) and Carer Support Needs Assessment Tool (CSNAT). Kendall’s tau-b correlation coefficient was calculated to determine the correlation between the NAT:PD-ILD patient well-being items and a subset of SGRQ-I similar constructs identified a priori. The prevalence and bias-adjusted kappa (PABAK), Cohen’s kappa and observed percentage agreement were used to assess agreement between the NAT:PD-ILD carer items and appropriate CSI and CSNAT constructs identified a priori. Data simulations indicated that a sample size of 65 patients would allow estimation of the kappa statistic for agreement such that the CI would not extend beyond the neighbouring category.

All analyses were conducted using Stata V.13 (StataCorp, Stata Statistical Software: Release 13. College Station, TX: StataCorp, 2013).

Results

Reliability

Fifty-three clinicians (32 doctors, 18 physiotherapists, 2 clinical physiologists, 1 nurse) participated in 64 first views across 10 videos, with 21 test–retest observations on four videos.

Inter-rater reliability

Eleven (69%) NAT:PD-ILD items reached at least fair agreement (weighted kappa >0.2).

Test–retest reliability

Five items exhibited at least moderate agreement (weighted kappa >0.4) (table 1).

Table 1

Inter-rater reliability test and test–retest results, weighted kappa

NAT:PD-ILD item	Inter-rater reliability					Test–retest*
	Number of observations	Distribution of categories†			Weighted kappa (SE)	% Agreement	Weighted kappa (SE)
		0	1	2
1. Does the patient have a carer?	77	NA	14%	86%	0.69 (0.24)	64.3	0.00 (–)
2. Has the patient or carer requested a referral to a Specialist Palliative Care Service (SPCS)?	70	NA	9%	91%	0.82 (0.44)	72.7	−0.14 (0.28)
3. Do you require the assistance of the SPCS?	65	NA	20%	80%	0.13 (0.30)	72.7	0.23 (0.29)
4. Is the patient experiencing unresolved physical symptoms?	89	6%	38%	56%	0.49 (0.21)	96.3	0.75 (0.22)
5. Is the patient experiencing unresolved psychological symptoms?	76	36%	57%	8%	0.32 (0.16)	81.7	0.00 (0.24)
6. Does the patient have problems with daily living activities?	85	8%	54%	38%	−0.07 (0.41)	79.0	−0.13 (0.19)
7. Does the patient have concerns about spiritual or existential issues?	57	79%	18%	4%	0.44 (0.30)	85.7	0.59 (0.24)
8. Does the patient have work, financial or legal concerns?	75	83%	13%	4%	0.15 (0.58)	‡	‡
9. From the health delivery point of view, are there health beliefs, cultural or social factors involving the patient or family that are making care more complex?	57	77%	21%	2%	−0.55 (0.73)	85.9	−0.03 (0.21)
10. Is the carer or family distressed about the patient’s physical symptoms?	85	32%	47%	21%	−0.17 (0.39)	90.8	0.65 (0.22)
11. Is the carer or family having difficulty providing physical care?	79	61%	34%	5%	0.59 (0.23)	81.6	0.10 (0.23)
12. Is the carer or family having difficulty coping with the patient’s psychological symptoms?	71	69%	30%	1%	0.36 (0.29)	94.4	0.64 (0.23)
13. Is the carer or family concerned about financial or legal concerns?	69	93%	7%	0%	0.77 (0.24)	‡	‡
14. Is the carer or family experiencing problems that are interfering with their functioning or interpersonal relationships, or is there a history of such problems?	67	61%	37%	1%	0.21 (0.38)	41.2	−0.15 (0.22)
15. Is the carer or family experiencing unresolved psychosocial problems or feelings that are interfering with their well-being or functioning?	70	47%	43%	10%	0.75 (0.12)	94.7	0.78 (0.23)
16. Is the carer or family experiencing grief over the future death of the patient?	45	69%	27%	4%	0.57 (0.27)	75.0	−0.03 (0.27)

*39/53 (74%) saw only one video consultation, 13 (25%) saw two video consultations and 1 (2%) saw four video consultations, though they may have watched each video up to two times (mean (SD) 1.3 (0.6), median 1).

†Items 1–3: 1=yes, 2=no; items 4–16: 0=none, 1=some/potential, 2=significant.

‡Same category assigned by every rater on both occasions; too few categories to calculate kappa statistic.

NA, not applicable; NAT:PD-ILD, needs assessment tool: progressive disease in interstitial lung disease.

Nine clinicians (six doctors, three nurses) and 68 patients (mean age 66, SD 10.3; 62% men; 45 with a carer (28 participated); 35% oxygen therapy; 56% interstital pulmonary fibrosis; 80% Medical Research Council breathlessness 3–5) were recruited (online supplementary eTable 1). The SGRQ-I mean summary score was 62.5 (SD 20.9): symptom component 67.4 (26.1); activities component 82.1 (23.2); impact component 51.2 (24.3). Scores for the carer comparator outcome measures are shown in online supplementary eTable 2. Agreement between NAT:PD-ILD concerns and comparator outcomes is shown in table 2. Items 2 (unresolved psychological symptoms/loss quality of life), 3 (problems with daily living activities), 5 (work, financial or legal concerns) and 6 (health beliefs, cultural or social factors) of the NAT:PD-ILD were significantly positively correlated with their comparator SGRQ-I scores (r range 0.24–0.36, P<0.05). PABAK values comparing the NAT:PD-ILD items with CSI/CSNAT items were mostly positive (0.04–0.57, minimum 52% agreement); however, items 11 and 13 have negative PABAK values (interpersonal relationships and grief topics).

Table 2

Relationship between NAT:PD-ILD items and SGRQ-ILD comparator items, and between the NAT:PD-ILD items relating to the carer and appropriate CSI and CSNAT items

Agreement between NAT:PD-ILD items and SGRQ-ILD comparator items			The prevalence and bias-adjusted kappa (PABAK), Cohen’s kappa and percentage of agreement between NAT:PD-ILD items relating to the ability and well-being of the carer and appropriate CSI and CSNAT items
NAT:PD-ILD Patient well-being ‘Does the patient have…’	The SGRQ-I comparator	Kendall’s tau-b (Kendall’s score, SE of score, P value)	NAT:PD-ILD; carer items	Comparator items from CSI and CSNAT	n	PABAK	Cohen’s kappa (SE)	% Agreed
4. Unresolved physical symptoms	Part 1 Q1–6; part 2 section 3 Q1–6; and part 2 section 4 Q6	0.16 (266, 163.0, P=0.10)	10. Distressed about the patient’s symptoms?	CSI8: Some behaviour is upsetting. CSI12: Feeling completely overwhelmed CSNAT3: Managing your relative’s symptoms, including giving medicines	28	0.57	0.53 (0.19)	79
5. Unresolved psychological symptoms/loss quality of life?	Part 1 Q6; part 2 section 4 Q1–6; and part 2 section 7	0.32 (512, 158.6, P=0.001)	11. Having difficulty providing physical care?	CSI3: It is a physical strain. CSI12: Feeling completely overwhelmed CSNAT3: Managing your relative’s symptoms, including giving medicines CSNAT5: Providing personal care for your relative CSNAT9: Equipment to help care for your relative CSNAT12: Practical help in the home	28	0.14	0.23 (0.14)	57
6. Problems with daily living activities?	Part 2 section 2 Q1–5; part 2 section 4 Q4+5; part 2 section 5 Q1–5; part 2 section 6 Q1–4; and part 2 section 7	0.36 (588, 159.7, P<0.001)	12. Having difficulty coping with the patient’s psychological symptoms?	CSI7: There have been emotional adjustments. CSI8: Some behaviour is upsetting. CSI12: Feeling completely overwhelmed	28	0.36	0.36 (0.18)	68
7. Spiritual or existential concerns?	Section 4 Q2; and part 2 section 7	0.11 (128, 129.9, P=0.33)	13. Concerned about financial or legal issues?	CSI7: There have been emotional adjustments. CSI8: Some behaviour is upsetting. CSI12: Feeling completely overwhelmed	27	0.41	0.31 (0.18)	70
8. Work, financial or legal concerns?	Part 2 section 1	0.34 (111, 53.2, P=0.04)	14. Experiencing problems that are interfering with interpersonal relationships or functioning, or is there a history of such problems?	CSI11: It is a financial strain. CSNAT4: Your financial, legal or work issues	28	−0.57	−0.13 (0.07)	21
9. Health beliefs, cultural or social factors making care delivery complex?	Part 2 section 4 Q1–6	0.24 (250, 108.6, P=0.02)	15. Unresolved psychosocial problems or feelings?	CSI4: It is confining. CSI5: There have been family adjustments. CSI6: There have been changes in personal plans. CSI7: There have been emotional adjustments. CSI9: It is upsetting to find…has changed so much from his/her former self. CSI10: There have been work adjustments. CSNAT2: Having time for yourself in the day CSNAT11: Talking with your relative about his or her Illness	27	0.04	0.17 (0.14)	52
–	–	–	16. Grief over the future death of the patient	CSNAT13: Knowing what to expect in the future when caring for your relative*	26	−0.46	−0.10 (0.10)	27

*This is more of a practical question, but was the nearest construct within the CSNAT.

Levels of concern in the NAT:PD-ILD were grouped into 0=‘None’ vs 1=‘Some/potential’ + ‘Significant’; the CSI responses were coded 0 for ‘No’ and 1 for ‘Yes’; and the four CSNAT item responses were categorised into two groups (0=‘No’ vs 1=‘A little more’ + ‘Quite a bit more’ + ‘Very much more’). For each NAT:PD-ILD item, the agreement between whether or not the carer indicated at least some problems and whether or not they had a ‘1’ in any of the recoded CSI or CSNAT items they were being compared with was calculated.

CSI; Carer Strain Index; CSNAT; Carer Support Needs Assessment Tool; NAT:PD-ILD, needs assessment tool: progressive disease in interstitial lung disease; PABAK; prevalence and bias-adjusted kappa; SGRQ-ILD, St George’s Respiratory Questionnaire-interstitial lung disease.

Discussion

Items within the NAT:PD-ILD demonstrated acceptable inter-rater reliability and construct validation given the broad constructs assessed and the breadth of clinical experience. The constructs of patient-reported quality of life (SGRQ-I) and assessment of need are related but different, therefore the relatively small number of items rated as moderate or strong is unsurprising. Similarly, many carer-related items both on NAT:PD-ILD and within CSI and CSNAT capture areas of concern that overlap, but are not directly comparable. The NAT-PD-C, with similar psychometric properties, resulted in reduced patient and carer needs when applied in practice; the key factor in any clinical tool.5

The NAT:PD-ILD is best seen as a communication and decision tool where action is thereby triggered if more in-depth exploration is needed, rather than an outcome measurement. The challenges of a clinical assessment and diagnosis are recognised as an inexact science with variation between clinicians.9 10 Study clinicians had a wide range of clinical experience to increase generalisability. Some NAT:PD-ILD items with poor agreement (inter-rater and construct) are consistent with the clinician participants’ expressed lack of confidence, for example, spiritual and existential concerns and may rather reflect an educational need.7

The clinician participants rated the videos after 10–15 min of training, inter-rater reliability is likely to improve with practice. Clinicians using the tool in daily practice will gain more experience as they use the tool.

Although we reached our target sample size for the construct validity analysis, we have insufficient sample for the carer comparisons. Also, our sample was convenience, not consecutive, potentially affecting representativeness.

The tool is yet to be tested in a clinical trial to evaluate its use by clinicians in terms of impact on patient and carer experience.

Conclusions

The adapted NAT:PD-ILD has adequate reliability and construct validation. Effectiveness in clinical practice, and optimum implementation are yet to be evaluated and identified.