Outcomes frequently specified in Cochrane reviews of community-based psychosocial interventions for adults with severe mental illness: A systematic search and narrative synthesis.

BACKGROUND: Outcome selection in intervention studies is a critical issue for synthesizing evidence. This study is aimed to investigate outcomes used in Cochrane reviews assessing community-based psychosocial interventions for adults with severe mental illness.
METHODS: Cochrane reviews that evaluated a community-based psychosocial intervention for adults with severe mental illness were searched electronically and manually. We extracted all outcomes specified in the Methods section in each Cochrane review. Outcomes that represent the same concept and context were synthesized into an outcome term. Outcome terms were categorized according to the existing taxonomy.
RESULTS: We included 33 Cochrane reviews. Of the 216 outcome terms identified, 13 were used in more than half of the reviews: quality of life, mental state, admission to hospital, economic outcome, leaving the study early, social functioning, satisfaction, global state, relapse, adverse events/effects, carer satisfaction, employment, and duration of admission. Most outcome terms were categorized into the life impact core area (55%), followed by the resource use area (21%).
CONCLUSIONS: Our study provides a candidate outcome list for developing a core outcome set for severe mental illness and offers a basis for comparison for future outcome investigation on mental health research.

INTRODUCTION

Measuring identical outcomes in clinical trials has become important internationally to synthesize evidence and achieve better health and social care. , This global trend has accelerated the development of core outcome sets (COSs) in the field of mental health. , , Currently, there are no consensus outcome sets for evaluating community‐based psychosocial interventions for people with severe mental illness, including schizophrenia.

Community‐based psychosocial interventions cover various diagnoses rather than one specific diagnosis, and include a wide variety of approaches, such as assertive community treatment, supported employment, and cognitive behavioral therapy. Despite the variety of diagnoses and interventions, there are two reasons to develop COSs for community‐based psychosocial interventions for people with severe mental illness. First, although their diagnoses differ, such individuals appear to have common needs and goals regarding living in the community despite serious functional impairment. , An outcome set that corresponds to common needs and goals would help researchers focus not just on alleviating symptoms specific to each disease, but also on people's problems and concerns in everyday life and their expectations concerning community mental health services. Second, there should be one outcome set that can be used to evaluate various interventions to assess how effectively each intervention achieves the end goal. People who use several types of community service may not necessarily have a completely different purpose for each service. It would be beneficial to define important outcomes independently of specific interventions in order to highlight the ultimate goals of people with severe mental illness.

COSs are generally developed through multiple steps. Researchers first create a long list of potentially essential outcomes by reviewing relevant studies and performing qualitative surveys of stakeholders, while also collaborating with service users. The purpose of assessing previous studies is to identify outcomes that have been commonly reported. An effective approach is to examine the outcomes used in relevant Cochrane reviews, which provide high‐quality evidence and have international influence. However, no studies have reported the results of this approach. We, thus, examined outcomes and their frequencies in Cochrane reviews of community‐based psychosocial interventions for people with severe mental illness. Identifying outcomes in Cochrane reviews can contribute to developing a long list of outcomes for future COS studies in this area and help researchers select outcomes when they conduct clinical trials.

METHODS

Cochrane reviews were searched electronically and manually. The electronic search was performed on September 6, 2021 using the Cochrane library database with relevant keywords (see Supporting Information S1; Table 1). Cochrane reviews were eligible if they evaluated a community‐based psychosocial intervention for adults with severe mental illness (ie, schizophrenia, psychosis, bipolar disorder, or major depressive disorder). Studies were excluded if they met any of the following criteria: (a) the intervention involved medication only or was implemented only in inpatient settings; (b) the participants were mainly children, adolescents, elderly people, or families of patients; and (c) the participants had postnatal or prodromal psychosis. We excluded children, adolescents, and elderly people since they are likely to have developmentally specific goals and needs that are qualitatively different from those of adults with severe mental illness. We did not limit the date of publication. Cochrane reviews that were considered out of date were also included since we focused on outcomes rather than results. Since electronic search often does not cover all relevant records, hand‐searching is generally recommended in a search process. To avoid missing potentially relevant records, we also searched the Cochrane library manually using the keyword “severe/serious mental illness” and obtained reviews that seemed to meet our criteria. Title and abstract screening and full‐text screening were independently completed by two authors (MI and SY). Any identified discrepancies were resolved through discussion with a co‐author (CF).

For each Cochrane review, we extracted all outcomes specified in the Methods section. More specifically, we identified outcomes that indicated what factor was measured rather than those describing how it was measured. For example, we extracted only “general behaviour” from “no clinically important change in general behavior.” Composite outcomes (eg, “quality of life/treatment satisfaction”) were divided into two or more outcomes, as appropriate (eg, “quality of life” and “treatment satisfaction”). We did not distinguish whether outcomes were originally classified as primary or secondary. In Cochrane reviews, outcomes are listed using headings, which refer to outcomes in a broad sense (eg, “Mental state”), and subheadings, which refer either to outcomes in a narrow sense or to how the outcomes are measured (eg, “Not any change in general mental state”). Outcomes were extracted regardless of whether they were presented as headings or as subheadings. An identical outcome was extracted only once from a single review. One author (MI) extracted outcomes and another author (TK) checked whether the outcome extraction was correct.

We synthesized outcomes according to the definition of “unique outcomes” by Young et al, which indicates that outcomes with different words, phrasing, or spelling that represent the same concept and context are recognized as one outcome. For example, “length of hospital stay” and “duration of admission” were combined as one outcome, namely, “duration of admission.” An outcome indicating the number of participants who met a condition (eg, “number of participants admitted to hospital”) was considered to be the same as an outcome indicating the proportion of participants (eg, “admission to hospital”). When there was a broader outcome (eg, “mental state”) and a narrower outcome (eg, “positive symptoms”), the narrower one was not included in the broader one but was organized separately. Outcome categorization was conducted according to the taxonomy developed by Dodd et al Additional subdomains were created if appropriate. Outcomes were synthesized and categorized based on discussion among the four authors, one of whom was a user researcher who have a diagnosis of major depressive disorder and an experience as a service user for about 15 years (MI, SY, TK, and MO). In this study, the user researcher critically examined the consistency of the categories and interpreted the results.

RESULTS

Electronic and manual searches identified 119 and 10 Cochrane reviews, respectively. Of these, 33 Cochrane reviews were included (see Supporting Information S1; Figure 1, Table 2). The year of publication ranged from 2001 to 2019. A total of 954 outcomes were initially extracted. The number of outcomes per review ranged from 6 to 54 (median 33; interquartile range 11). After integrating outcomes with the same wording, the number of outcomes was 491. These were synthesized into 216 unique outcome terms (see Supporting Information S2).

Thirteen outcome terms (6%) were specified in more than half of the reviews (Table 1). Six of the 13 outcome terms were categorized into the life impact area. On the other hand, nearly half (n = 99, 46%) of the outcome terms were specified in only one review, and about 70% (n = 150, 69%) were specified in three or fewer reviews.

TABLE 1

Outcome terms specified in more than half of the included Cochrane reviews

Outcome term	Number of Cochrane reviews which specified the outcome term	%	Core area
Quality of life	30	90.9	Life impact
Mental state	29	87.9	Physiological/clinical
Admission to hospital	29	87.9	Resource use
Economic outcome	27	81.8	Resource use
Leaving the study early	27	81.8	Life impact
Social functioning	26	78.8	Life impact
Satisfaction	26	78.8	Life impact
Global state	25	75.8	NA
Relapse	24	72.7	Physiological/clinical
Adverse events/effects	21	63.6	Adverse events/effects
Carer satisfaction	19	57.6	Life impact
Employment	18	54.5	Life impact
Duration of admission	18	54.5	Resource use

The outcome terms were categorized into five core areas with 38 outcome domains according to Dodd et al. The life impact area included the largest number of outcome terms (n = 119, 55%); these involved aspects of functioning or satisfaction in real life, such as physical/social/role/emotional/cognitive functioning, quality of life, and satisfaction with services. All 33 reviews specified more than one life impact outcome term. The other core areas were death area (n = 4, 2%), physiological/clinical area (n = 40, 19%), resource use area (n = 45, 21%), and adverse events/effects area (n = 4, 2%). The three most frequently specified domains were delivery of care (n = 29; life impact), psychiatric outcomes (n = 23; physiological/clinical), and hospital/service use (n = 23; resource use). Four outcome terms (global state, behavior, specific behavior, and general functioning) could not be categorized into any domain because their concepts are unspecific.

DISCUSSION

In this study, 13 outcome terms were specified in more than half of the reviews. This is highly consistent with the report of Wuytack et al, which found that nine outcomes were used in more than half of the 10 Cochrane reviews published by the Cochrane schizophrenia group in 2013. Seven outcomes were included in both studies: quality of life, mental state, economic outcomes, leaving the study early, social functioning, satisfaction, and adverse events/effects. On the other hand, six of our 13 frequently specified outcome terms were not among the nine outcomes identified by Wuytack et al. Given that we focused on psychosocial interventions in the community, carer satisfaction and employment may reflect the characteristics of our target population, as its members are likely to live with carers or intend to get a job in the community. In other words, the difference in outcome terms between the two studies may be attributed to the targeted interventions and populations.

Adverse events/effects and economic outcomes might have ranked high because this study investigated the outcomes of Cochrane reviews. The Cochrane Handbook of Systematic Reviews of Healthcare Interventions suggests that every healthcare intervention comes with some risk of harmful or adverse events/effects, and recommends that all Cochrane reviews try to include some consideration of the adverse aspects of the interventions. Economic perspectives are also recognized as useful in Cochrane reviews, in order to build evidence on the cost‐effectiveness of optimal healthcare decision‐making. Our results confirmed that Cochrane reviews emphasize the importance of the safety and feasibility of interventions by assessing adverse events/effects and economic outcomes.

In terms of outcome categorization, the life impact area included the majority of outcomes (n = 119, 55%) in this study. Six of the 13 most frequently specified outcome terms were also in the life impact area. These results differ from those of a previous study that categorized outcomes reported in clinical trials of adolescent major depressive disorder. In that study, the majority of outcomes (62%) were in the physiological/clinical area, while only a relatively small number (27%) were classified into the life impact area. This difference in proportion implies that researchers who evaluate interventions for severe mental illness in community settings are more interested in outcomes related to various aspects of patients’ functioning or satisfaction in life than in clinical indicators.

Our study is the first to comprehensively investigate the outcomes specified in Cochrane reviews of community‐based psychosocial interventions for people with severe mental illness, and to report these outcomes along with their frequencies. Since Cochrane reviews are internationally influential systematic reviews, the outcome data used by Cochrane review authors impact future randomized controlled trials (RCTs) and systematic reviews. They also offer a basis for comparing future outcome studies that investigate outcomes used in studies of other designs (such as RCTs or systematic reviews other than Cochrane reviews), mental illnesses, or populations.

A particularly important contribution of our study is that it provides a list of candidate outcomes for the development of a COS for community‐based interventions for severe mental illness. Collecting potential outcomes by reviewing previous studies is an essential step in COS development. An organized outcome list with a well‐documented creation process can be a promising tool to develop a COS for people in the community with severe mental illness. Note that we do not insist that researchers use the popular outcomes in Cochrane reviews for intervention studies. Our list of outcomes may be used in the initial stage of COS development, but the following stages need to include patients and other stakeholders to build a consensus on outcomes that should be assessed in intervention studies.

Limitations

This study focused only on the methods sections of Cochrane reviews and did not examine the results sections. To determine what outcomes are of interest to researchers conducting RCTs, future studies may need to assess the outcomes reported by individual RCTs. We did not extract outcomes from protocols of Cochrane review or contact Cochrane Groups to identify ongoing reviews or updates. Therefore, there may be other relevant reviews that were not included in this study.

CONCLUSIONS

Our comprehensive list of outcomes in Cochrane reviews can be used as a candidate outcome list for developing a COS for people in the community with severe mental illness. Future COS development should synthesize patients’ and other stakeholders’ views with scientific accumulation, such as our list, to build a consensus on important outcomes in this area.

CONFLICT OF INTEREST

The authors declare that they have no conflict of interest.

AUTHOR CONTRIBUTIONS

MI led the analysis and interpretation of data and drafted the first version of the manuscript. SY designed the study concept, contributed to the analysis and interpretation of data, and supervised the entire process. TK contributed to the analysis and interpretation of data. MO and CF contributed to the interpretation of data. SS helped search for Cochrane reviews by hand. All authors critically revised the manuscript and approved the final version of the manuscript. Other contributions are stated within the article.

Supporting Information S1

Click here for additional data file.

Supporting Information S2

Click here for additional data file.