Measuring the success of interventions for caregivers: a focussed systematic review.

PURPOSE OF REVIEW: The selection and application of outcome measures are fundamental steps in the research process because they inform decisions around intervention efficacy. We aimed to describe the outcomes used in trials of supportive interventions for adult caregivers of people with three exemplar life-limiting conditions: heart failure, dementia or stroke. RECENT
FINDINGS: We performed a focussed review of 134 published trials that included interventions designed to improve caregivers' health and wellbeing and/or ability to function in their caring role. We extracted and categorized all caregiver outcomes described in the studies. We identified inconsistency in the outcomes measured; frequent use of bespoke and adapted tools (29% of outcomes were bespoke), and a lack of clarity in outcome priorities (the mean number of outcomes per trial was four [range: 1-11]). Outcome scales that purport to measure the psychological impact of the caring role were the most popular tools in all three caregiver groups.
SUMMARY: Outcomes used in trials related to adult caregivers are characterized by inconsistencies in outcome measure selection and assessment. This heterogeneity complicates comparisons of treatments and attempts to pool data.

INTRODUCTION

Informal caregivers play an essential role in health and social care systems. Taking the United Kingdom as an example, there are approximately 6.8 million unpaid caregivers [1-3] and this number is set to rise to 9 million by 2037 [4]. Improvements in healthcare coupled with financial restraint on health and social care funding continues to increase the demand for caregivers and their contribution to society. It is estimated that the annual economic value provided by caregivers in the United Kingdom is £132 billion (comparable to total annual UK health spending) [5].

There is ambiguity around the preferred terminology for caregivers, with terms such as carer, informal caregiver, unpaid carer and support person, among others often cited in literature. For the purposes of this review, a caregiver is defined as a person who assists at least one other person to carry out their activities of daily living to varying extents depending on need and capability. Although they may volunteer their time, they are not volunteers [6]. We selected the term caregiver as preferred terminology to reduce possible confusion with the job title of carer, which relates to a person who is employed to care for another.

no caption available

OUTCOME MEASURES IN CAREGIVER RESEARCH

Improvements to treatment and management of life-limiting conditions have resulted in the role of the caregiver becoming increasingly enduring and complex [7▪▪]. The pressures and unpredictability of the role can have significant consequences on the caregiver's physical and mental health and cause financial and domestic hardship [8]. In response, there is now an extensive body of research dedicated to developing and testing interventions to support caregivers. To measure the efficacy of these interventions, researchers are required to select appropriate outcome measures. Outcome measures should help bridge the gap between participants’, researchers’ and clinicians’ often different expectations and priorities. Outcome assessment can take many forms, for caregiver research patient/person-reported outcome measures (PROMs) may have relevance. PROMs set out to measure the impact of disease and interventions on aspects of daily living such as physical function, mental health, socialization and wellbeing [9]. They are usually short, self-assessed questionnaires that are completed (at least) before and after an intervention to measure an aspect of the individual's life at that point in time [10].

There are methodological, ethical and financial considerations when selecting outcome measures for potentially vulnerable groups, such as caregivers. Researchers must select outcome measures that pose the least risk to the participants, as any adverse reactions experienced by participants may put the cared-for person at risk. The role of the caregiver is dynamic, and it can be challenging to separate changes in the caregivers’ state from changes in their loved one's condition. Using a well-validated tool is the best way to ensure research findings are coherent and accurate regardless of the perceived complexity of the intervention and/or outcome measures [11,12].

Heterogeneity in the assessments used as study outcome has been demonstrated in many areas of clinical research, including stroke [13], dementia [14] and cardiology [15]. This inconsistency in outcome assessment complicates comparative or pooled analysis and can be a barrier to research progress.

We sought to describe the outcomes used in caregiver researcher, with an emphasis on three exemplar chronic conditions; stroke, dementia and heart failure. These conditions also represent the archetypes of caring roles, including symptoms management (heart failure), physical disability (stroke) and cognition (dementia). Hence, their inclusion maximizes the potential to capture a range of outcome measures that reflects the diversity of caring.

METHODS

Our primary objective was to describe the outcomes used in trials of supportive interventions for adult caregivers of people living with heart failure, dementia or stroke. We used a focussed review approach, adhering to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines for reporting where relevant (Fig. 1). We registered our protocol (review registry: 437).

FIGURE 1

PRISMA flowchart of results. PRISMA flowchart of results from the time-limited CINAHL, Medline and PsycInfo database searches. CINAHL, Cumulative Index to Nursing and Allied Health Literature; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses.

Search strategy

It was not our intention to provide a comprehensive review of all caregiver intervention studies. Rather, we devised a search strategy to give an overview of the field. This approach is in keeping with previous reviews of population-specific outcome assessments [16-18]. The Cochrane database was the primary source of trial extraction. Cochrane systematic reviews were obtained by manually searching review group databases. We screened titles and abstracts for studies that included caregivers as participants and/or caregiver outcomes. As our interest was outcomes used in published trials, data obtained from conferences, unpublished studies and personal correspondence were excluded.

Recognising that Cochrane reviews may not include contemporary studies, we complemented the Cochrane database search with a time-limited search (01/01/15–31/12/17) of Cumulative Index to Nursing and Allied Health Literature (CINAHL) (EBSCO), Medline (OVID) and PsycInfo (EBSCO) databases (Appendix 1 for search syntax).

Selection criteria

We included trials that quantitively measured caregiver outcomes from interventions that aimed to improve adult caregivers’ physical and psychological wellbeing and/or their ability to function in their caring role. We excluded trials that did not report caregiver-specific outcomes, even if caregivers participated in the trials. We also excluded qualitative studies and trials where the intervention was respite care for the cared-for person (because the caregiver is not the active participant in receipt of this intervention).

Titles and abstracts were screened with the full-text assessment of relevant results (M.D.), using Covidence (Veritas Health Innovation, Australia). Results were discussed with experienced systematic reviewers (B.J. and T.Q.), with disagreements resolved through discussion until consensus was reached.

Data extraction and synthesis

We extracted data on intervention, primary caregiver outcomes and secondary or uncategorized caregiver outcomes. The data extraction process was carried out by M.D. and discussed with B.J. and T.Q. when guidance was required.

We classified each outcome assessment tool in relation to the construct it was attempting to measure. The construct classifications were created by scoping the categorized outcome measures:

‘Psychological experience’ was used when the tool was attempting to measure an experience or symptom of caregiver mental ill-health, such as depression or anxiety.

‘General health’ was used when the tool was attempting to measure aspects of overall wellbeing but also included aspects of physical health.

‘Coping’ was used when the tool measured physical and/or psychological aspects specific to caring in one tool, for example the caregivers perceived control over their caring role, or the quality of relationship between the caregiver and cared-for person.

‘Social support’ was used when the use of community support and/or social services, or the caregivers support network (engagement with friends, family and so on) were measured.

‘Disease knowledge’ was used when the tool attempted to measure the caregivers understanding of an aspect of the condition.

We also categorized each outcome measurement tool into validated or bespoke categories:

A validated tool was one that had undergone previous validity and reliability testing.

A bespoke outcome measure was one that was described by the study authors as being designed for that study, or where the researchers used an adapted version of a previously validated tool to fit their study.

We grouped interventions into six categories for ease of interpretation because of the heterogeneity in intervention terminology. Again, this was done by scoping the results. Categories included:

‘Psychotherapy’ typically included a cognitive behavioural intervention or counselling.

‘Education/training’ interventions usually involved training the caregiver (and sometimes the cared-for person) in an aspect of carrying out the caring role (such as coping with problematic behaviours) or disease management (such as oral care).

‘Case management’ was used when the intervention was explicitly stated or implicitly described as case/care management.

‘Support (or enhanced support)’ typically blended more than one style of intervention with the aim of supporting the caregiver to continue in their caring role.

‘Alternative’ interventions included spiritual, mindfulness and reminiscence interventions.

‘Exercise’ interventions involved teaching the caregiver exercises that they could mediate with the cared-for person.

FINDINGS

Please refer to Figures 2 and 3 and Tables 1 and 2 for a detailed overview of the intervention types and outcomes yielded from the 134 trials included in this review. Overall, 29% of outcome measures were bespoke (n = 79/266). The mean number of outcomes per trial was four (range: 1-11). There was substantial variation between diagnosis groups in the average number of outcome measures and the tools used, psychological experience-based outcomes appeared among the most popular tools in all three groups (Appendices 2–4 for detailed results of included trials, interventions and outcomes; refer to Appendix 5 for the abbreviations used in the tables of results).

FIGURE 2

Intervention types by diagnosis. Visual display of the results of the categorized intervention types grouped by diagnosis of the cared-for person.

FIGURE 3

Total number of outcomes per study. This box and whisker diagram is an overview of the distribution of the number of outcomes per study, grouped by diagnosis of the cared-for person. It includes the range of outcomes, median number and the quartiles for each diagnosis group.

Table 1

Overview of included studies

	Dementia	Stroke	Heart failure
Total no. of studies	91	32	11
Total no. of outcomes	176	76	22
Median number of outcomes per triala	4.4	3.3	2.7
IQR range	3	3.25	2
Most used outcome measure tool(s)	CES-D n = 30	Short Form-36 n = 9	BDI n = 2; Short Form-36 n = 2; CES-D n = 2; FAD n = 2
No. of bespoke measure tools (% of total number of outcomes)	55 (31%)	20 (26%)	4 (18%)

aSignificant difference, P = 0.01.

IQR, interquartile range.

Table 2

Construct measures of all outcomes including primary outcome measures

	Dementia	Stroke	Heart failure
Coping	67 n = 25	19 n = 3	9 n = 2
Psychological experience	45 n = 21	17 n = 1	9 n = 1
General health	29 n = 4	13 n = 1	3 n = 1
Disease knowledge	5 n = 2	10 n = 1	1 n = 0
Social support	21 n = 0	17 n = 1	1 n = 1
Miscellaneous	2 n = 3	0 n = 0	0 n = 0
Studies with no primary caregiver outcome(s) or unspecified outcomes	36	25	6

n = Number of outcomes that were list as primary outcomes.

Trials relating to the caregivers of people with dementia represented the largest group, with 91 trials (68% of total trials included). This group had the largest variation in number of outcome measures, with 176 outcome measures used and 139 of those only used in one study. The most commonly used tool was the Centre for Epidemiology Studies – Depression Scale (CES-D) (n = 30 trials). However, there were seven other tools that measured depression or depressive symptoms, most of which were used in only one trial. The second most-used tool was the Zarit Burden Inventory (ZBI) (n = 23 trials). The Revised Memory and Behaviour Problem Checklist (RMBPC) was used in 13 trials and had the most versions, with a modified version and its subscales also used. The caregivers of people with dementia were the only group that also included outcome measures obtained from laboratory testing or physical examinations (Table 3) and miscellaneous measures related to finance and safety.

Table 3

Physical examination or laboratory testing outcomes

Test	Outcome measure	Number of trials
Salivary cortisol	Stress	3
Catecholamines	Influence hypertension and ischemic heart disease	1
Plasma cortisol	Stress	1
Plasma renin	Stress	1
Plasma aldosterone	Stress	1
Actigraph monitor	Sleep quality	1
Plasma biomarkers	Stress	1
Heart rate	Stress	1
Blood pressure	Stress	1

Of the 32 studies that related to caregivers of stroke survivors, there were 76 different tools used to measure caregiver outcomes. Fifty-seven (75%) had been previously validated, with the remaining measures developed for their specific trial or adaptions/subscales of previously validated tools. The most popular tool used was the Short Form-36 (n = 9 trials), used three times as a primary outcome. The CES-D was used in seven trials but only once as a primary outcome. The caregiver strain index (CSI) and the Hospital Anxiety and Depression Scale (HADS) were used six times. The CSI was not listed as a primary outcome in any of the studies and the HADS was listed as a primary outcome twice. Disease knowledge was measured by 10 tools, six of which were from bespoke questionnaires, making disease knowledge the most commonly measured construct by a bespoke measure.

Of the 11 studies that related to caregivers of people with heart failure, four tools (Short Form-36 Health Survey, Beck Depression Inventory (BDI), CES-D and Family Assessment Device Questionnaire) were used twice, with all other tools used once. Most [18] of the tools that were used were validated, with one study using three bespoke measures. Depression was the most commonly measured outcome, with four tools used to measure this (BDI-II, HADS and CES-D), all of which are validated. Quality of life was measured twice, using the Short Form-12 and Short Form-36. Five trials had an explicit primary outcome listed.

DISCUSSION

We have identified inconsistencies in many aspects of measuring outcomes in caregiver intervention research (Fig. 4). These inconsistencies included:

FIGURE 4

Selecting appropriate patient/person-reported outcome measures (PROMs). A flowchart to help researchers undertaking research where PROMs may be appropriate outcome measures in their outcome selection process.

Outcome selection

Outcome implementation (including what they purported to measure)

The use of bespoke and adapted/modified tools

Outcome prioritization

The prevalence of outcome measures with a psychological focus suggests some agreement in direction of supportive interventions for caregivers. However, there was a difference of opinion in what aspect of psychological impact should be measured (burden, strain and depression all among the most popular outcomes). The prioritization of psychological impact from caring is understandable because of the enhanced risk of caregivers experiencing depression, anxiety, stress and physical injury when caring for someone with a life-limiting condition [19]. However, heterogeneity remained among the tools used to measure psychological consequences. For example, caregiver burden, a widely accepted complex experience that often occurs as part of caregiving was measured using 10 different tools.

Outcome selection

The heterogeneity in outcomes is also seen in outcomes used in the trials looking at specific diseases [16-18]. Such heterogeneity suggests that researchers have not established clear priorities and objectives for intervention trials related to life-limiting conditions. This may suggest that the people involved (the cared-for person and their caregivers) are not adequately consulted during the research process.

Moniz-Cook et al.[20] recommend that outcome measures for caregivers involved with dementia research include psychological experience, burden, coping with behaviour and quality of life (QOL). The specific tools that are recommended include the CES-D and HADS for anxiety and depression, the General Health Questionnaire for general distress and ZBI for burden, with more research required for QOL measures [20]. In our review, there was some consensus for QOL tools as the Short Form-12/36 was the most popular tool in all three caregiver groups.

A structured review of PROMs for the Department of Health [16] recommends that the EuroQolEQ-5D , which measures QOL, is an appropriate measure for stroke research. However, this recommendation is in relation to the stroke survivor and not necessarily their caregiver. The EQ-5D was cited in four studies within the stroke survivors’ caregiver group. Qualitative interviews [17] with stroke survivors suggest that caregiver burden should also be a priority.

There is a dearth of literature concerning PROMs in the field of heart failure research. The few existing reviews did not find caregiver outcomes as a focus nor did they recommend their increased consideration or inclusion [18,21]. However, Chang et al.[22] reviewed outcomes that are important to people with heart failure and found that caregiver burden, functional status, symptom management and survival were all priorities. Yet, not all outcomes are of equal value to the caregiver and cared-for person. Priorities may also change as the disease progresses. This caregiver group also yielded the smallest amount of results from the Cochrane and database searches, which suggests an urgent need for more heart failure caregiver intervention research and reviews of the existing studies.

Some of the variances identified could be addressed using predetermined outcome measures. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative is a database of literature related to core outcome sets (COS) that aim to specify an agreed minimum set of outcome measures, with consideration given to ‘what’ and ‘how’ this is carried out (http://www.comet-initiative.org/). Although COMET does not have a caregiver-specific COS, recommendations for caregiver outcome measures can be found within associated COS literature. Developing a COS specific to caregivers presents challenges. Measures must be psychometrically robust while still capturing the nuances involved with caring for someone with a life-limiting condition [23]. Furthermore, it is important when working with potentially vulnerable groups, such as caregivers of people with life-limiting illnesses, that the measures are appropriate and safeguard participants. Owing to these ethical and practical considerations, outcome measures that are selected should prioritize quality of life and family support [24]. However, these challenges should not be insurmountable; instead, they highlight the importance of developing COS specific to caregivers of people with life-limiting conditions.

Outcome implementation

There was ambiguity around the application and interpretation of certain outcome measure tools. The selection of tools and the concepts they are suitable for measuring is of fundamental importance in intervention trials. Inconsistent or inappropriate outcome measures could invalidate results and consequently, prevent the development of future trials with effective interventions [13].

As an example, the RMBPC is a valid and reliable tool that measures problematic behaviours typically associated with caring for someone with dementia, and their reactions/feelings to them. However, as an outcome, it was used to measure other constructs such caregiver burden, distress and appraisal. Caregiver reactions to problematic behaviour are one aspect of caregiving that may contribute to the challenges of the caring role. However, concepts like caregiver burden and caregiver appraisal (which includes satisfaction and mastery [25]) are far greater than this one potentially challenging aspect to caring.

Coping was a popular construct measure in all caregiver groups. However, it was broadly defined compared with others. Outcome measures that related to coping ranged from how prepared the caregiver felt about undertaking their caring role (preparedness for caregiving scale), to their desire to institutionalize the cared-for person (desire to institutionalize bespoke measure), to self-assessment of their competence in the role (Perceived Competence Scale). It is understandable that coping might be a necessary measure for caregiver intervention trials; if an intervention can enhance a caregiver's ability to cope with their role then it is much more likely that the cared-for person will be well tolerated and remain in their preferred place for as long as possible. However, there are validated tools to measure the explicit concept of coping, such as the Revised Ways of Coping Checklist [26]. Yet this tool was only used in one trial in its entirety, and a subscale used in one other.

A degree of inconsistency across disease groups is understandable. For example, the most commonly used tools in the studies related to caregivers of stroke survivors were the Caregiver Burden Scale. However, the ZBI was the most commonly used measure of burden among trials for caregivers of people with dementia. The ZBI was developed specifically for caregivers of people with dementia [27] and the Caregiver Burden Scale has shown good validity and reliability when used to measure burden among caregivers of stroke survivors [28].

Bespoke and adapted/modified outcomes

The use of bespoke/adapted outcomes was noted across all caregiver groups. In most studies, the authors gave a description of their bespoke tool. However, there was very little explanation why they opted to develop their own rather than using an established reliable and validated tool. This also applies to the use of adapted and subscale tools that were used in many of the trials included in this review. There may be concerns in outcome measure selection for potentially vulnerable groups and, therefore, there may be temptation to develop one that appears to fit better. However, outcome measures should not replace the therapeutic relationship; instead, they should be used alongside it to instil a person-centred approach [11]. There may also be concerns regarding the suitability of older tools when there is new understanding and knowledge in a field of study, such as the management of life-limiting conditions. However, such concerns regarding the relevance of existing outcome measures should be based on a thorough assessment of their use to inform decisions around potential misuse and reliability [15]. Adhering to this process will identify when there is a need for the development of new outcome measures in a transparent and justifiable way.

Outcome prioritization

Many of the studies did not clearly specify which outcomes were of primary interest to the researchers and often reported many disparate measures. Although richness of data is laudable, the interpretation of study results becomes difficult when many, potentially conflicting, results are presented with no clear indication on the primary outcome of interest. It is recognized that researchers can change the emphasis of a study report, focussing on positive secondary outcomes rather than the neutral primary outcome and for this reason publication of a trial protocol that clearly defines primary and secondary outcomes is now considered best practice. Ultimately, the absence of clear primary outcomes dilutes the findings and reduces the likelihood that they will inform clinical practice [15].

Poor prioritization of outcome measures also raises questions about whether the participant's time has been effectively used. In many instances, more than one outcome measure seemed to be assessing the same construct within a trial. This is of importance when working with caregivers who are taking time away from caring for people with life-limiting illnesses, and potentially palliative and end-of-life care needs [12].

STRENGTHS AND LIMITATIONS

We acknowledge that the selection of three life-limiting conditions does not cover the breadth of caregiver or life-limiting conditions research. Instead, selecting these conditions allowed us to develop a focussed review. We also acknowledge the limitations associated with having one person carry out the literature searches, data collection and data synthesis. However, data analysis and reporting of the collected results were undertaken between all reviewers. It was anticipated that the review of trials included in Cochrane systematic reviews would be the primary source of study retrieval. However, more articles were retrieved from the time-limited, database searches. Because the findings were categorized by the diagnosis of the cared-for person, most of the retrieved studies are related to dementia trials. Therefore, any findings from this review may be more applicable to dementia trials, than stroke, heart failure and other life-limiting conditions.

CONCLUSION

It is in the interest of clinicians, policymakers and the public to support caregivers of people with life-limiting actions to fulfil their caring role. Developing, testing and implementing supportive interventions that assist them is just one way that this can be done. The development of caregiver-specific COSs in collaboration with caregivers would ensure that outcome selection in caregiver intervention research is robust, reliable and valid. This would also ensure that outcomes for caregiver research can be consistent and transparent. However, consistency in outcome selection and measurement does not remove person-centredness. This is because, by their very nature, PROMs allow participants to express their perceptions of their experiences in a way that can be understood and translated by clinicians and researchers.

Acknowledgements

None.

Financial support and sponsorship

This work is supported and part-funded by NHS Greater Glasgow & Clyde.

The systematic review formed the first stage of a PhD currently being undertaken by M.D. which is funded as a Studentship Grant by NHS Greater Glasgow and Clyde. T.J.Q. is supported by a joint Chief Scientist Office/Stroke Association Senior Clinical Lectureship.

Conflicts of interest

There are no conflicts of interest.

REFERENCES AND RECOMMENDED READING

Papers of particular interest, published within the annual period of review, have been highlighted as:

▪ of special interest

▪▪ of outstanding interest