Test-Retest Reliability of the Short-Form Survivor Unmet Needs Survey.

OBJECTIVE: Reliable and valid needs assessment measures are important assessment tools in cancer survivorship care. A new 30-item short-form version of the Survivor Unmet Needs Survey (SF-SUNS) was developed and validated with cancer survivors, including hematology cancer survivors; however, test-retest reliability has not been established. The objective of this study was to assess the test-retest reliability of the SF-SUNS with a cohort of lymphoma survivors (n = 40).
METHODS: Test-retest reliability of the SF-SUNS was conducted at two time points: baseline (time 1) and 5 days later (time 2). Test-retest data were collected from lymphoma cancer survivors (n = 40) in a large tertiary cancer center in Western Australia. Intraclass correlation analyses compared data at time 1 (baseline) and time 2 (5 days later). Cronbach's alpha analyses were performed to assess the internal consistency at both time points.
RESULTS: The majority (23/30, 77%) of items achieved test-retest reliability scores 0.45-0.74 (fair to good). A high degree of overall internal consistency was demonstrated (time 1 = 0.92, time 2 = 0.95), with scores 0.65-0.94 across subscales for both time points.
CONCLUSIONS: Mixed test-retest reliability of the SF-SUNS was established. Our results indicate the SF-SUNS is responsive to the changing needs of lymphoma cancer survivors. Routine use of cancer survivorship specific needs-based assessments is required in oncology care today. Nurses are well placed to administer these assessments and provide tailored information and resources. Further assessment of test-retest reliability in hematology and other cancer cohorts is warranted.

Introduction

Lymphoma blood cancers are malignant T or B cell lymphocytes in the lymphatic system and are categorized under two main types: non-Hodgkin lymphoma (NHL) and Hodgkin Lymphoma (HL). NHL represents approximately 88% of all lymphomas, while HL is predominately diagnosed in the adolescent and young adult population.[1] Combined, they represent the sixth most common cancer diagnosis worldwide.[1] Consistent with worldwide trends, the incidence of lymphoma in Australia is increasing, and with a projected diagnosis of 6232 cases in 2017, this equates to 4.6% of all cancer cases.[2] An estimated mortality rate of 1481 equates to 3.1% of all deaths from cancer in 2017.[2] Projected figures for 2017 in the USA have a similar projected incidence of lymphoma of 4.8% and mortality of 3.6%.[1] Treatment for lymphoma generally comprises high-dose chemotherapy and/or targeted immunotherapy agents and may include radiotherapy and hematopoietic stem cell transplants.[3] These treatments have resulted in an improvement to overall survival of approximately 76% at 5 years compared with 52% at 5 years in the 1980s.[2] Notwithstanding the positive impact treatment has had on survival rates,[4] the consequences of disease and treatment continue long after treatment completion.[5] Long-term and late effects may produce ongoing unmet needs such as fear of recurrence, fatigue, poor nutrition, exercise, fertility, relationship, financial, employment, and insurance issues.[6]

To provide optimal supportive cancer care to lymphoma survivors, the identification of patients’ perceived concerns and level of support needed is required.[5] This is especially important for younger patients (18–45 years of age) where the expectation of long-term remission can raise additional concerns and unmet needs.[7] Receiving relevant information and practical support soon after treatment ends, especially resources related to healthy lifestyle behaviors,[7891011] can help mitigate the impact of disease and treatment and lead to fewer unmet needs further along the survivorship continuum.[1213] A qualitative study with lymphoma cancer survivors (n = 17) undertaken in Western Australia[14] reported unmet informational and practical needs as participants transitioned from treatment to the survivorship phase. The findings suggested tailored post-treatment support and interventions are fundamental components of excellent survivorship care.

The measures used to assess unmet needs are equally important. Generic cancer measures which comprise items related to diagnosis and treatment are often not specific enough for the survivorship phase.[15] Comprehensive, relevant, reliable, and validated needs assessment measures that are survivor-specific are essential to capture unmet needs that become evident when treatment ends.[15] These measures can guide health professionals in providing individualized information, support, and resources.[515] Two recent systematic reviews[1516] revealed that needs assessment tools are varied and may not capture all the possible unmet needs patients may have. The reviews likewise found validity and reliability evidence limited. The Survivor Unmet Needs Survey (SUNS) was identified as a measure that had strong psychometric properties and was developed and psychometrically tested with a large cross-sectional sample of cancer survivors (n = 550) including a small cohort of hematology cancer participants (n = 31, 5.6%).[17] Campbell et al.[17] confirmed a high overall internal consistency of items for their study with an overall Cronbach's alpha of 0.99. The authors also reported high test–retest reliability although the results were not published.[17] Internal consistency of the SUNS was further tested in two studies of hematological cancer survivor cohorts. A cross-sectional study with 529 hematological cancer survivors[18] demonstrated overall Cronbach's alpha values >0.9, and a weighted Kappa coefficient score of >0.6 for test–retest reliability; acceptability was reported for 40/89 (45%) items. Qualitative data from 17 semi-structured interviews indicated that the SUNS was considered relevant by this cohort of hematological cancer survivors.[18] A cross-sectional study of hematological cancer survivors from Australia and Canada (n = 437) reported similar levels of unmet needs across the two cohorts using the SUNS, with fatigue (n = 76, 17%) and financial concerns (n = 39, 9%) rated as high unmet needs.[9] Despite the clinical utility of the original SUNS, it was considered potentially burdensome for use in the clinical setting given the large number of items (n = 89). In 2014, the 30-item short-form-SUNS (SF-SUNS) was developed and validated with a mixed sample of cancer survivors (n = 1589), including hematological cancer survivors (n = 84, 5%).[5] Construct validity and intraclass correlation coefficients (ICCs) of the SF. were similar to those of the original SUNS. Cronbach's alpha scores for the final four domains were ≥0.85, and ICCs for the three domains from the original SUNS (financial concerns, information, and access and continuity of care) and the SF-SUNS were high (>0.9). Discriminant validity demonstrated the SF-SUNS ability to discriminate between individuals who had recently received treatment and those who had not. The authors recommended further testing of the SF-SUNS for test–retest reliability.[5] The 30-item SF-SUNS was therefore judged to be more practical and likely to be completed by participants in our larger study, particularly as the SF-SUNS was one of four instruments to be administered to participants in a pilot randomized trial to measure the effect of a nurse-led survivorship model of care.[19]

For researchers and clinicians to develop targeted follow-up support for cancer cohorts underrepresented in survivorship literature, such as lymphoma,[20] cohort-specific studies in the early survivorship phase are required.[21] Therefore, this study recruited only those with a lymphoma diagnosis who had completed treatment. Discerning the issues and concerns of this group requires survivor-specific measures that are psychometrically sound and fully tested. The SF-SUNS has been used within the clinical setting; however, since test–retest reliability of the SF-SUNS had not been established, the aim of the present study was to establish test–retest reliability of the SFSUNS to add to the psychometric data available in the published literature on this instrument.

Methods

Design

Test–retest reliability of the SF-SUNS was conducted at two time points: baseline (time 1) and 5 days later (time 2). This time frame was chosen to reduce recall bias and change in the level of unmet needs.[22] Ethical approval to conduct the study was obtained from the human research ethics committee of the study site (2015-020) and university (015007F).

Population and setting

A convenience sample of 40 lymphoma cancer patients who were 3 months’ posttreatment completion were recruited from the hematology department of a large tertiary hospital in Western Australia. Inclusion criteria were pathologically confirmed new diagnosis of NHL or HL; completed first-line curative intent chemotherapy or second-line curative intent autologous stem cell transplant within the previous 3 months; no radiological evidence of lymphoma posttreatment (on positron emission tomography [PET] scan); able to understand and read English; and over 18 years of age. Participants were excluded if they had not been treated with chemotherapy; had received further treatment at another hospital (as experiences or interventions may have introduced bias); or were cognitively impaired or experiencing an acute mental health condition that prohibited the provision of informed consent.

Sample size

The sample size calculation was derived from Walter et al.[23] and used a fixed alpha of 0.05 from two observations with reliability values of R0 = 0.6 (acceptable) and R1 = 0.8 (expected), indicating a minimum sample size of n = 39.

Short-form Survivor Unmet Needs Survey

The SF-SUNS assesses unmet needs across four domains: information needs (3 items); work and financial needs (8 items); access and continuity of care needs (6 items); and coping, sharing, and emotional needs (13 items). Patient self-reported concerns and the level of support required are measured using a Likert-type scale: 0 – no unmet need, 1 – low unmet need, 2 – moderate unmet need, 3 – high unmet need, and 4 – very high unmet need. Domain scores are generated by adding each item score and dividing by the total number of domain items.[24]

Procedure

The researcher identified and approached eligible participants after treatment completion to discuss the study and provide them with a participant information and consent form. Following informed consent, demographic and baseline (time 1) SF-SUNS questionnaires were then administered to participants. After completion of the questionnaires, participants were provided with another blank copy of the SF-SUNS accompanied by instructions to complete the questionnaire at home 5 days later and postback using the supplied reply-paid addressed envelope. Participants were advised to record the date of completion if this differed from the specified due date.

Data collection

At the request of the research team's hematologist, baseline demographic and SF-SUNS data were collected from consenting participants 3 months posttreatment completion and PET scan to confirm the absence of disease. Demographic information obtained included lymphoma type, stage of disease, type of treatment received (chemotherapy +/− radiotherapy), date of diagnosis, time since diagnosis, comorbid conditions, gender, age, weight, marital status, age of children (if any), postcode, occupation, income level, education level, and health behaviors such as smoking and alcohol consumption. Participants then completed the SF-SUNS at time 2 (5 days following time 1 completion) at home.

Data analysis

All analyses were performed using IBM SPSS Statistics Version 25 data analysis software (IBM Corp. Released 2017. IBM SPSS Statistics for Macintosh, Version 25.0. Armonk, NY: IBM Corp). Descriptive statistics were used to analyze all data. Descriptive analyses were used to analyze and describe demographic data. To assess for absolute consistency of SF-SUNS items for test–retest reliability data, an ICC with a random-effects model was used to compare each item at time 1 and time 2. The ICC measure was chosen for its ability to discriminate between sets of scores ranked in the same order but not necessarily in agreement and adjusts for the degree of test–retest agreement expected by chance.[2526] The closer the value of the ICC to 1.0, the greater the reliability of the item or measure.[27] The guidelines developed by Cicchetti and Sparrow[28] were used to determine the level of clinical significance of the ICC values obtained: <0.40 = poor, 0.40–0.59 = fair, 0.60–0.74 = good, and >0.75 = excellent. For this study, items classified as achieving “fair to excellent” reliability, ICC >0.40,[29] were reported. Cronbach's alpha, a measure of internal consistency, was used to measure the scale reliability.

To examine the distribution of unmet needs, the five levels of unmet need were collapsed to three levels. A score of 0 (no unmet need) remained the same. Scores of 1 or 2 (low and moderate unmet need) were reclassified as 1 (low–moderate unmet need), and scores of 3 or 4 (high and very high unmet need) were classified as 2 (high–very high unmet need).

Results

Participant characteristic

There were slightly more male (n = 22, 55%) participants, and a greater number of participants with NHL (n = 29, 72.5%) compared with HL (n = 11, 27.5%) [Table 1]. This was in keeping with the current disease statistics which reflect a greater number of NHL than HL diagnoses.[1] Almost one-third of participants were aged between 18 and 39 years (32.5%), and a greater proportion had a university qualification (n = 16, 40%) [Table 1]. Although the majority of participants were currently working (n = 15, 37.5%) and had been throughout their treatment, 30% (n = 12) were looking for work or had no return to work date set. Over half the participants had a partner (n = 25, 62.5%). Forty participants completed both time 1 and time 2 SF-SUNS. The majority of participants (n = 35, 87.5%) completed time 2 SF-SUNS 5 days after time 1 (range 4–7 days).

Table 1

Baseline participant demographic and disease characteristics (n=40)

Characteristics	n (%)
Gender
Male	22 (55.0)
Female	18 (45.0)
Age group (years)
18-39	13 (32.5)
40-59	12 (30.0)
60-74	9 (22.5)
75+	6 (15.0)
Marital status
Single	10 (25.0)
Married/de facto	25 (62.5)
Divorced	3 (7.5)
Widowed	2 (5.0)
Lymphoma diagnosis
Non-Hodgkin	29 (72.5)
Hodgkin	11 (27.5)
Highest level of education
Secondary school or less	11 (27.5)
Trade, vocational college	13 (32.5)
University or higher	16 (40.0)
Employment status
Working	15 (37.5)
Retired	13 (32.5)
Looking for work/no return to work date	12 (30.0)

Test–retest

ICCs, 95% confidence intervals, and clinical significance are shown in Table 2. One (3%) item met the “excellent” criteria for clinical significance; Finding car parking I can afford at the hospital or clinic. Twelve (40%) items met the “good” criteria (0.60–0.74) and 11 (37%) items met the “fair” criteria (0.40–0.59). In summary, test–retest data showed “fair” to “good” reliability for the majority of items (23/30, 77%).

Table 2

Item test-retest reliability and internal consistency (n=40)

Domain (n=4)	Item description	ICC (95% CI)	Level of clinical significance	Cronbach's alpha		Item-to-total correlation
				Time 1	Time 2	Time 1	Time 2
Information needs	Items (n=3)			0.74	0.69
	Finding information about complementary or alternative therapies	0.69 (0.49-0.83)	Good			0.30	0.50
	Dealing with fears about cancer spreading	0.56 (0.30-0.74)	Fair			0.59	0.63
	Dealing with worry about whether treatment has worked	0.57 (0.32-0.75)	Fair			0.65	0.71
Work and financial needs	Items (n=8)			0.65	0.83
	Worry about earning money	0.63 (0.40-0.79)	Good			0.49	0.47
	Having to take a pension or disability allowance	0.39 (0.09-0.62)	Poor			0.45	0.38
	Paying household bills or other payments	0.69 (0.49-0.83)	Good			0.55	0.60
	Finding what type of financial assistance is available and how to obtain it	0.70 (0.50-0.83)	Good			0.67	0.71
	Finding car parking that I can afford at the hospital or clinic	0.76 (0.59-0.86)	Excellent			0.02	0.45
	Understanding what is covered by my medical insurance or benefits	0.31 (0.01-0.57)	Poor			0.20	0.06
	Knowing how much time I would need away from work	0.74 (0.55-0.85)	Good			0.55	0.50
	Doing work around the house (cooking, cleaning, home repairs, etc.)	0.37 (0.07-0.61)	Poor			0.12	0.70
Access and continuity of care	Items (n=6)			0.89	0.85
	Having access to cancer services close to my home	0.45 (0.16-0.66)	Fair			0.44	0.62
	Getting appointments with specialists quickly enough (oncologist, surgeon, etc.)	0.38 (0.08-0.61)	Poor			0.70	0.44
	Getting test results quickly enough	0.66 (0.44-0.81)	Good			0.57	0.51
	Having access to care from other health specialists (dietitians, physiotherapists, occupational therapists)	0.53 (0.26-0.72)	Fair			0.51	0.67
	Making sure I had enough time to ask my doctor or nurse questions	0.58 (0.33-0.75)	Fair			0.59	0.48
	Getting the health care team to attend promptly to my physical needs	0.53 (0.26-0.72)	Fair			0.59	0.50
Coping, sharing and emotional needs	Items (n=13)			0.90	0.94
	Telling others how I was feeling emotionally	0.43 (0.14-0.65)	Fair			0.58	0.48
	Finding someone to talk to who understands and has been through a similar experience	0.33 (0.02 to 0.58)	Poor			0.45	0.57
	Dealing with people who expect me to be “back to normal”	0.62 (0.39-0.78)	Good			0.57	0.77
	Dealing with people accepting that having cancer has changed me as a person	0.51 (0.24-0.71)	Fair			0.68	0.81
	Dealing with reduced support from others when treatment has ended	0.67 (0.46-0.81)	Good			0.82	0.82
	Dealing with feeling depressed	0.73 (0.55-0.85)	Good			0.53	0.72
	Dealing with feeling tired	0.49 (0.21-0.69)	Fair			0.57	0.71
	Dealing with feeling stressed	0.55 (0.29-0.74)	Fair			0.78	0.69
	Dealing with feeling lonely	0.72 (0.52-0.84)	Good			0.53	0.61
	Dealing with not being able to feel “normal”	0.47 (0.20-0.68)	Fair			0.57	0.70
	Trying to stay positive	0.63 (0.40-0.79)	Good			0.55	0.65
	Coping with having a bad memory or lack of focus	0.64 (0.41-0.79)	Good			0.50	0.86
	Dealing with changes in how my body appears	0.28 (−0.04-0.54)	Poor			0.23	0.24

ICC: Intraclass correlation, CI: Confidence interval

Internal consistency

Overall Cronbach's alphas were 0.92 at time 1 and 0.94 at time 2, with subscales [Table 2] ranging from 0.74 and 0.69 for information needs, 0.65 and 0.83 for work and financial needs, 0.89 and 0.85 for access and continuity of care, and 0.90 and 0.94 for coping, sharing, and emotional needs, respectively. These results support strong internal consistency for the overall scale. Item-to-total correlations between 0.40 and 0.70 indicate that items are not redundant or measuring needs similar to other items within the instrument.[30] Using this criterion, the SF-SUNS demonstrated item-to-total correlations between 0.40 and 0.70 at time 1 for 24 items (80%) and at time 2 for 19 items (63%) [Table 2]. The majority of items were considered relevant and to be measuring unique needs.

Discussion

Our study is the first to report test–retest data for the SF-SUNS. The majority of items met absolute consistency for reliability ICC scores of >0.40 for test–retest, categorized as “fair” to “good.” An “excellent” clinical significance score was achieved for only one item (3%), related to car parking costs which are unlikely to change over time. Needs-based instruments such as the SF-SUNS measure the degree of an individual's perceived unmet need at one point in time. Importantly, Cronbach's alpha scores at time 1 and time 2 demonstrated a high degree of internal consistency and high item-to-total correlations, confirming that items in the tool were reliable.

A criterion for psychometrically sound needs-based tools is the requirement for an instrument to be responsive to changes over time.[313233] Although our ICC results may reflect the responsiveness of the SF-SUNS to changes in need over the data collection period, further research is required to detect clinically meaningful change for patients.[16] All participants completed the time 2 questionnaire at home, well away from the hematology clinic where the time 1 questionnaire was completed. It is possible that participants may have had additional time to more accurately reflect on the level of unmet need. Similarly, time 1 scores may have been impacted by participants’ anxiety at the hospital appointment where patients often worry about test results and potential relapse.[34] In addition, fatigue is a recognized effect of lymphoma treatment[7] and may have potentially affected participant responses at either time point. Finally, most items were similarly balanced for both time points from “no unmet need” to “low unmet need” or “low unmet need” to “no unmet need.”

It is important to allow cancer survivors the opportunity to self-identify unmet needs and issues of concern. Survivorship needs-based instruments provide a consistent method for this purpose.[35] Furthermore, it is important that any tool is responsive to change as individuals’ issues, concerns, thoughts, and feelings can change from day-to-day,[3233] particularly during survivorship transition as individuals move on with their lives after cancer treatment. Such reliable and valid instruments can facilitate individualized survivorship care and tailored support and resources.[15]

It is important to note that the original SUNS demonstrated low test–retest reliability acceptability,[18] with the authors suggesting that the test–retest timeframe was too long at 28 days. Since our study was part of a larger study involving an intervention group, a 5-day later test–retest assessment was deemed an appropriate timeframe to ensure completion of the time 2 SF-SUNS before the implementation of any needs-based interventions associated with the larger study.[19] Importantly, this time period was also in keeping with the recommended 2–14-day time period for test–retest procedures.[313233]

A limitation of this study may have been the sample size of 40 participants, despite sample size calculations indicating that this number would be sufficient to adequately perform test–retest reliability with confidence. Many participants (n = 16, 40%) attended the baseline appointment, where time 1 SF-SUNS was administered, accompanied by a support person (partner or family member). We acknowledge that this may have influenced time 1 responses. Likewise, time 2 responses may have similarly been influenced as the SF-SUNS was completed at home. We can confirm that participants did not receive any needs-based interventions between time 1 and time 2 completion of the SF-SUNS.

Conclusion

We suggest that needs-based assessments should be used routinely during the survivorship period to facilitate survivorship care that is tailored and responsive to individuals’ changing needs. Valid and reliable survivor-specific measures are essential for routine screening and follow-up. Nurses in particular are a valuable resource in the survivorship phase to assess for areas of concern or unmet needs and for the provision of information, support, and resources that are tailored to the individuals’ unique needs. Further testing of the SF-SUNS is recommended in hematology and other cancer populations to further understand and demonstrate the responsiveness of this instrument to changes in need over the survivorship period.

Financial support and sponsorship

This research is supported through the University of Notre Dame Australia's Collaborative Research Network Grant and an Australian Government Research Training Program Scholarship.

Conflicts of interest

There are no conflicts of interest.