Consensus development of priority outcome domains for community mental health cares by multiple stakeholders: Protocol for an online Delphi study in Japan.

BACKGROUND: Treatment goals for mental illness have expanded from hospital discharge and improved functioning to employment, living alone, and personal realization. These changes in treatment goals have also influenced mental health research. Recent studies have addressed the development of core outcome sets focusing on clinical aspects of mental illness such as depression and anxiety. However, a well-developed framework of essential outcomes for people with mental illness (service users) who live in the community is lacking. In addition, recent worldwide trends suggest more patient and public involvement and the importance of considering multiple stakeholders' views in the area of mental health research. Purpose of this study is to explore consensus on high-priority outcome domains among multiple stakeholders in community mental healthcare fields in Japan.
METHODS: A three-step approach to developing an outcome list will be used. First, we developed a long list of outcomes for community mental health through a literature review, focus group interviews with key stakeholders, and online questionnaire surveys of service users and caregivers. Second, the long list was checked and revised in a pilot study. Third, the long list will be shortened to the outcome list through the Delphi methodology with participation from multiple stakeholders. DISCUSSION: Identifying important common outcome domains through collaboration with multiple stakeholders appears to contribute to the development of evidence for community mental health research in Japan. In addition, the study process itself may help promote patient and public involvement in education, practice, and research in the field of community mental health.

BACKGROUND

Proper outcome selection is essential for the research and political contexts in the community mental healthcare setting. Given the recent spread of evidence‐based policy making, reasonable outcome selection for multiple stakeholders in each culture has become increasingly important.

In Japan, the number of service users who live in the community has gradually increased through the deinstitutionalization of patients with mental illness. Accordingly, the treatment goal for mental illness has been shifting to hospital discharge and improved functioning, as well as independent living and internal self‐realization. Changes in treatment goals have influenced mental health research. For example, in addition to clinical outcomes (eg, duration of admission, readmissions, symptoms, functioning, etc), there has been growing interest in measuring outcomes related to personal recovery. As the outcomes of interest increase and diversify, researchers have been seeking a consensus on outcome selection. This is occurring for the community mental health setting in Japan.

Some international projects on outcome consensus had been launched. For example, the Core Outcome Measures in Effectiveness Trials (COMET) Initiative was established to address the issues of inconsistent outcome selection and bias in outcome reporting. The Core Outcome Set (COS) has been developed in each clinical area. The COMET Initiative identified the need for core outcome sets across a wide range of health disciplines and their role in coordinating this information. In addition, the International Consortium for Health Outcomes Measurement (ICHOM) was established in 2012 to redefine the value of health care through patient‐centered outcomes. It has started to develop a standard set of outcomes, placing importance on the participation of patients themselves. In summary, the process of outcome selection has received increased attention. In the field of psychiatry, previous studies have developed core outcome sets that should be assessed in research studies. For example, COMET and ICHOM have developed core outcome sets focusing on clinical aspects of mental illness such as depression, anxiety, and bipolar disorder.

While a core set of psychiatric outcomes has been developed, there are some considerations when developing priority outcome domains for community mental health care. First, community mental health service providers generally serve people with severe mental illness who have a variety of diagnoses and do not always offer diagnosis‐specific services. Community mental health now assumes that have a significant capacity to manage their own illnesses and to pursue personally goals, even people with the most severe mental illnesses or ongoing symptoms. Current mental health trends and interventions thus emphasize self‐management, self‐agency, strengths, resilience, and capacity for functional recovery. In other words, community mental health service users have similar needs, goals, and interests regardless of their diagnosis. Second, community mental health services are usually comprehensive and provided by a multidisciplinary team rather than through a single intervention method. For example, while a staff member in a community mental health service team provides employment services, another member provides cognitive behavioral therapy. Therefore, priority outcome lists that are diagnosis‐specific or only focus only on a specific intervention type may not be applicable to community mental health care. In addition, this study will be conducted only with stakeholders in Japan. The reason for restricting this study to participants living in Japan is that the priority outcome domains in the field of community mental health care may differ from culture to culture because of differences in the nature of disability welfare services and the perception of mental illness in each country.

Insufficient involvement of multiple stakeholders is another concern for outcome selection in community mental health care. The outcomes that researchers and patients have rated as important have not always been the same. , In addition, the involvement of only service users is inadequate for selecting reasonable outcomes. A proper outcome selection process for community mental health care requires involvement from diverse stakeholders such as caregivers, service providers, and government staff. The lack of involvement and an outcome selection framework may lead to irrelevant outcome measures, which may be ultimately a barrier for accumulating appropriate evidence about community mental health services. , Indeed, a recent worldwide trend suggests that the concept of patient and public involvement (PPI) and incorporation of the views of service users and other stakeholders is important in mental health research. In addition, PPI is a useful approach not only for intervention studies but also for outcome selection research.

The above limitations based on recent trends and features of community mental health care suggest that the issue of outcome selection or priority outcome domain. However, few studies have addressed culture‐specific consensus building for outcome selection, comprehensive outcome selection that does not focus on a particular diagnosis or intervention, or multiple stakeholder involvement in community mental health services. To address this evidence gap, we will conduct a Delphi survey with multiple stakeholders such as service users, caregivers, service providers, national and regional government staff, and researchers in Japan. The participation of diverse stakeholders will enable the identification of important common and diverse outcomes for community mental health care in Japan.

OBJECTIVES

The purpose of this study is to explore consensus on important and high‐priority outcome domains for multiple stakeholders in community mental health care and relevant research in Japan.

METHODS

Overview

A three‐step approach to identify the priority outcome domains will be used. First, a draft comprehensive list of outcomes was compiled through a review of outcomes reported in Cochrane reviews and existing randomized controlled trials, focus group interviews with key stakeholders, and online surveys of service users and caregivers. Second, the comprehensive outcome list was refined through a pilot study. Third, a Delphi study will be conducted using the comprehensive outcome list to develop outcome lists (Figure 1). Our research team includes researchers with backgrounds in academic medicine, nursing, social work, occupational therapy, music therapy, clinical psychology, and service user experience. The multidisciplinary research team has conducted or will conduct all the stages of this study and report the findings. In other words, all the process of the study has involved or will involve user‐researchers.

FIGURE 1

Research process

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. This study protocol was approved by the Tokyo Metropolitan University Ethics Committee (No. 20083) and the Research Ethics Committee of the National Center of Neurology and Psychiatry (No. A2021‐005). The study protocol was registered in the University Hospital Medical Information Network (UMIN) Clinical Trials Registry (UMIN‐CTR ID, UMIN000044680).

Step 1: Development of a long list of outcomes that are considered important by service users, caregivers, and stakeholders in Japanese community mental health research

To inform the design of the Delphi surveys, the sections on Steps 1 and 2 describe what we have already done. However, no substantive findings are presented in this paper. During Step 1, we developed a long list of outcomes in community mental health using the results from focus group interviews and online questionnaires, in addition to the data collected through a literature review. (Further details about extract process of outcomes are available in a Supporting Information 1 to this report.)

To form an accessible Delphi survey, the outcomes needed to be grouped together where there were areas of commonality or duplication and mapped into outcome domains. Four research meetings were held to remove duplication of outcome items, further consolidate common outcome items, and map outcome domains. Participants in these meetings were from a range of health and social care research backgrounds (doctors, nurses, psychiatric social workers, occupational therapists, clinical psychologists, and experts‐by‐experience). In addition, research team members (MI and TS) held multiple meetings to discuss and organize outcome items. All outcome items collected were maintained on an Excel spreadsheet. Research team members laid out the outcome items printed on paper on a table and summarized similar outcome items or deleted outcome items that were determined to be inconsistent with the purpose of this study. Any disagreements were resolved through discussion. Finally, 94 outcome items were identified (eg, Psychiatric symptoms or mental state; Physical health; Activity or leisure; Employment, life satisfaction, quality of life, or well‐being) and included in the Delphi study. We defined each outcome item and created a summary to understand the meaning of the items.

Step 2: Pilot study of the outcome list

We conducted a pilot study for the purpose of determining the adequacy of the long list and descriptions for each outcome item developed in Step 1. The pilot study also aims to obtain feedback on wording, usability, and visuality for the online survey form.

Participants and recruitment

Participants in the pilot study were service users, caregivers, community mental health service providers, government staff, and community mental health researchers. We recruited two or more participants from each group.

Data collection and analysis

In the pilot study, two rounds of online questionnaire surveys were conducted using a website.

We sent the participants an invitation to the survey website by e‐mail. We asked them to respond to the survey from the dedicated webpage. Potential participants were read the explanation of the study at the beginning of the webpage and then answered only if they agree to participate in the study.

The long list of outcomes generated during the Step 1 were presented to participants. Participants rated each outcome item in the long list on a seven‐point scale from "very important" to "not important at all." In addition, participants could write comments if they have opinions on each outcome item. They could also suggest new outcome items. Furthermore, the usability of the survey website and the clarity of the descriptions were evaluated.

We compiled and summarize the participants’ comments. Results were presented as averages, percentage of respondents with each score, and variance so that participants can confirm the variation in others’ opinions. In order to avoid the risk of adjustment bias by our research team, all comments were reflected in the summary. Corrections to wording and additions of items to the long list were made after discussion among research team members.

We reported the results of the first round to the participants. After confirming the results of the first round, participants were responded to the revised long list of outcomes.

Participants rated each outcome in the revised long list on a seven‐point scale from "very important" to "not important at all." Participants entered and submitted free‐text descriptions for each outcome.

Using the results of the pilot survey, we modified outcome items. Finally, 96 outcome items draft the "Long list of outcomes for community mental health research in Japan" to be used in the Delphi survey (Table 1).

TABLE 1

Long list of outcome domains

1.	Psychiatric symptoms or mental state	49.	Stigma and discrimination
2.	Psychological distress (anxiety or depression)	50.	Religion or beliefs
3.	Relapse or remission	51.	Sexual satisfaction
4.	Insight	52.	Cognitive functioning
5.	Substance use	53.	Knowledge of illness and services
6.	Self‐harm	54.	Medication adherence
7.	Violence or aggression	55.	Treatment adherence
8.	Suicidal ideation or attempt	56.	Attitudes toward medication or treatment
9.	Death—suicide	57.	Satisfaction with services
10.	Death—all causes	58.	Unmet needs
11.	Laboratory measures	59.	Perceived coercion
12.	Physical health	60.	Housing stability
13.	Weight and obesity	61.	Earnings
14.	Physical fitness	62.	Duration of stay in community / Duration of admission
15.	Chronic pain	63.	Family relationships or functioning
16.	Self‐care	64.	Living with family
17.	Perceived stress	65.	Having a role model
18.	Subjective health status	66.	Therapeutic relationship
19.	Overall functioning	67.	Costs of mental health care
20.	Daily living skills	68.	Costs of all care
21.	Contact with the legal system	69.	Mental health service use
22.	Interpersonal relations	70.	All types of hospital admission
23.	Communication skills	71.	Involuntary hospital admission
24.	Help‐seeking	72.	Involuntary treatment
25.	Partner or marriage	73.	Outpatient visits
26.	Overall social functioning	74.	Emergency service use
27.	Independent living	75.	Non‐psychiatric service use
28.	Social connectedness	76.	Number of caregivers needed to maintain stable state
29.	Activities or leisure	77.	Medication prescription
30.	Place of safety and belonging	78.	Adverse or side effects
31.	Competitive employment	79.	Caregivers' mental health
32.	All types of employment	80.	Caregivers' physical health
33.	Job matching the preferences of service users	81.	Caregivers' subjective health status
34.	Work‐related skills or vocational ability	82.	Family's stigma and discrimination
35.	Work tenure	83.	Caregivers' problem‐solving or coping skills
36.	Job hunting and related activities	84.	Caregivers' self‐esteem
37.	Childbirth and childcare	85.	Caregivers' knowledge of illness and services
38.	Caregiving for family members	86.	Caregivers' service satisfaction
39.	Education	87.	Caregivers' life satisfaction or quality of life
40.	Role in society	88.	Expressed emotions for caregivers
41.	Peer support	89.	Caregivers' service use
42.	Life satisfaction, quality of life, or well‐being	90.	Social support for caregivers
43.	Motivation	91.	Caregiver's perception of the care of the service user
44.	Empowerment or self‐determination	92.	Burden of care
45.	Self‐esteem	93.	Financial burden of care
46.	Resilience	94.	Influence on caregivers' lifestyles
47.	Feeling dependent on psychiatric treatments	95.	Influence on caregivers' jobs
48.	Symptom control skills or coping	96.	Influence on caregivers' leisure activities

Step 3: Delphi study

An online Delphi survey will be conducted to develop the priority outcome list for the community mental health setting in Japan. Since there are no definitive or optimal methods for conducting a Delphi survey, we will determine the survey methods accordingly with reference to previous studies.

Number of study participants: The Delphi survey for five panel attributes will include service users, caregivers, community mental health service providers, national or regional government staff that work in mental health, and community mental health researchers. While there is no clear criterion for the number of participants in a Delphi study, studies usually include 50 members per panel, according to a relevant review. With reference to previous studies, the number of expected participants in this study will be 250, consisting of 50 people for each panel attribute.

Response rate: In general, a response rate over 70% is required to obtain reliable results in Delphi studies. This study will follow previous studies and aim for a response rate of 70% or higher for each round.

Number of iterations for consensus building: Given the risk of participant burden and thus a lower response rate, a Delphi study in the field of health services generally needs two to three rounds for consensus building. , Based on previous studies, this study will include a total of three rounds.

Criteria for consensus: The definition of consensus in a prior study was 70% or more of respondents rating an outcome item as "important" or "very important”. In a review article, the basic definition was set in the range of 55%–100%, and since 70% was reported the most of using as criterion. The definition of consensus for this study is 70% or more. In addition, we will exclude items if 75% or more of the respondents rated them as “not important” or “not important at all” based on a past core outcome study of discharge interventions among service users.

Participants and recruitment

The potential participants of the Delphi survey will be stakeholders in community mental health care, as in the pilot surveys. There will be 50 stakeholders of each type. Considering an 11.7% attrition rate (about 6 participants) based on a previous study, we will recruit approximately 60 participants in each stakeholder group. Inclusion criteria include (1) ability to respond to the study voluntarily and (2) living in Japan. Exclusion criteria include (1) currently hospitalization, (2) age under 20 years, and (3) adult guardianship.

Other inclusion criteria will be established for each attribute panel in this study. The inclusion criteria for service users are as follows: (1) diagnosis of mental illness (potential participants with dementia and intellectual disability are excluded from participation), (2) disability that affects daily life (serious mental illness), and (3) use of community mental health and social services. The inclusion criteria for caregivers are as follows: (1) primary caregiver for a service user. The inclusion criteria for community mental health service providers are as follows: (1) national license in medical health and welfare that have (doctor, nurse, psychiatric social worker, occupational therapist, clinical psychologist, and other) and (2) history of involvement in community mental health and welfare services. This study will recruit at least five people from each major mental health profession (doctor, nurse, psychiatric social worker, occupational therapist, clinical psychologist) in order to reduce the occupational bias of participating supporters. The inclusion criteria for national or regional government staff are as follows: (1) working for a government agency and (2) involvement in work related to community mental health and social services. The inclusion criteria for researchers are as follows: (1) doing research in the field of community mental health care. We also asked whether the researcher held a national license in medical health and welfare. As in the community mental health service providers, we will recruit at least five people from each mental health profession (doctor, nurse, psychiatric social worker, occupational therapist, clinical psychologist, and other) to participate in the study.

We will recruit participants by explaining the purpose and content of this study to services user or caregiver organizations, professional associations, and academic societies. We will explain the survey outline to key persons in each organization. If the organizations or key persons agree to collaborate on the Delphi survey, they will inform their fellow members about this survey. We will also ask the participant to focus group interview participants and authors of the original papers on community psychiatry and community mental health services.

Delphi round 1

In the first round, participants will be asked to register online. Each participant will be asked to identify the applicable stakeholder group. A unique identifier will be assigned to each participant to allow identification of individuals that complete each round. Potential participants who are informed about the Delphi study via announcements from associations or emails from the research team will visit a webpage introducing the Delphi study web page of Department of Community Mental Health & Law (https://www.ncnp.go.jp/nimh/chiiki/research/32.html). If an individual is interested in participating in the Delphi study, they will provide an email address and obtain an invitation URL for the survey website. As in the pilot study, participants will read the detailed explanation of the study at the beginning of the webpage. They indicate agreement to study participation by participating in the survey. The long list identified through Steps 1 and 2 will be presented to participants. Participants will rate each outcome item on a seven‐point scale from "very important" to "not important at all." In addition, participants can submit comments for each outcome item. They can also suggest new outcome items if they think that there are essential outcome items that were not in the long outcome list.

Round 1 data analysis

The response rate will be assessed at the end of Round 1. The total number of respondents completing the round will be compared to the number of respondents who agreed to participate in this study. Response rate will be calculated by the number of participants overall and by stakeholder group (service user, caregiver, community mental health service providers, government staff, and researcher). For each outcome item, the distribution of scores will be summarized and analyzed. For each outcome item in Round 1, the proportion of participants scoring 1‐2, 3‐5, and 6‐7 on the seven‐point Likert scale will be calculated for each item. In addition, the results will be presented using averages, percentages of respondents with each score, and variance so that variations in opinions can be confirmed. Each outcome will be classified as “consensus in,” “consensus out,” or “no consensus.” “Consensus in,” referring to consensus that the outcome should be included in a core outcome set, will be defined as greater than 70% of participants scoring 6‐7 and less than 25% of participants scoring 1‐2. “Consensus out,” referring to consensus that the outcome item is not appropriate for include in the priority outcome domain, will be defined as greater than 75% of participants scoring 1‐2 and less than 50% of participants scoring 6‐7 in all stakeholder groups. “No consensus” is defined as any other distribution of scores. Only outcomes identified as "no consensus" will be re‐evaluated in Round 2.

All free‐text comments will be reflected in the summary to avoid the risk of adjustment bias introduced by the research team. Modifications to the wording of items or addition of outcomes based on free‐text comments will be made after discussion among the research team members.

Delphi round 2

We will report the results of Round 1 to the participants. After confirming the Round 1 results, participants will respond to the revised questionnaire. Participants will rate each outcome item on a seven‐point scale from "very important" to "not important at all." In addition, if participants have any comments on each outcome item, they could write them in the free‐text field.

Round 2 data analysis

The response rate will be assessed at the end of Round 2 using the same procedure as in Round 1. In Round 2, changes in the number of respondents from Round 1 will also be assessed. For each outcome item, the number of respondents and distribution of scores will be summarized and analyzed. For each outcome in Round 2, the proportion of participants scoring 1‐2, 3‐5, and 6‐7 on the seven‐point Likert scale will be calculated. In addition, the results will be presented using averages, percentages of respondents with each score, and variance so that the participants can understand the various views of others. Based on the consensus criteria, only those outcomes that are identified as "no consensus" will be re‐evaluated in Round 3.

Delphi round 3

We will report the results of Round 2 to the participants. The participants will confirm the Round 2 results. Next, they will respond to the revised questionnaire. As with Round 2, participants will rate each outcome items on a seven‐point scale to indicate their priority regarding the outcome items in the list.

Round 3 data analysis

At the end of Round 3, the response rate and number of respondents will be assessed using the same procedure as in the previous rounds. The number of respondents and distribution of scores will be summarized and analyzed for each outcome item. Round 3 data will be analyzed by stakeholder group. Results on outcome items will be presented using averages, percentages of respondents with each score, and variance. Each outcome will be classified as “consensus in,” “consensus out,” or “no consensus.”

Based on the results of Rounds 1‐3 of the Delphi survey, the outcomes categorized as “consensus in” by participants will be on the “Important outcome domain list for multiple stakeholders in community mental health research in Japan.”

DISCUSSION

Clarifying what are important outcomes is an essential process for developing effective services and policies. This study aims to develop consensus on key outcome domains for community mental health care among service users, caregivers, service providers, national and regional government staff, and researchers in Japan. The priority outcome domains identified in this study might contribute to proper outcome selection in future research. This study will identify the outcome areas that each stakeholder group considers to be high‐priority outcome domains for community mental health in Japan. This helps increase the understanding of common or disparate interests of research outcome domains between stakeholders. While international trends have emphasized the importance of PPI in the research context, few studies have addressed this issue in Japan. We involved a user‐researcher in all stages of the study process. For example, we adopted an idea from a user‐researcher for develop a web system that participants will be able to temporarily store their answers in the middle of a research. This system allows participants to take a break and participate in the survey. It might reduce the burden of the survey. In addition, the pilot study follows the process of modifying the wording from the preliminary survey so that participants from any attribute panel can easily understand the survey process and the contents of the Delphi survey. The collaborative process of this study itself might contribute to promotion of PPI in a Japanese community mental health service setting.

This study protocol has several strengths. First, the long outcome list will be developed using multiple strategies, including a literature review, focus group interviews, and web‐based surveys. This process enables the creation of a comprehensive list of outcomes for community mental health care. Second, the protocol of the Delphi survey clearly shows the criteria for consensus building. While there are various approaches to conducting a Delphi survey, this protocol paper has established the criteria as much as possible by referring to previous studies.

This study has some potential limitations. First, the use of an online survey method for the Delphi study restricts participation to individuals who have access to a computer and the internet. Thus, our study does not reflect the views of those who cannot use access the internet or a computer for any reason. Second, this study will only identify the outcome items with relatively high interest. In other words, this study will present the outcome domains considered to be important by key stakeholders, but it will not suggest particular scales or measurement methods for specific outcomes. For example, if the participants consider psychiatric symptoms as an important outcome, we will not mention any particular scale (eg, Positive And Negative Syndrome Scale or Brief Psychiatric Rating Scale).

Despite some limitations, this study has several benefits. The development of the outcome domain list through collaboration with multiple stakeholders will make the common important outcomes more visible. This domain list is expected to contribute to the development of evidence for community mental health research in Japan in future. Moreover, this research process highlights the importance of involving service users and caregivers and joint decision‐making, not just the interests of researchers. This study process will help promote PPI in the field of community mental health care in Japan.

CONFLICT OF INTEREST

The authors declare that they have no conflict of interest.

AUTHOR CONTRIBUTIONS

TS is the principal investigator responsible for the initial draft of this manuscript and organizing and implementing the study. TS, SY, MO, MA, TK, MI, NY, and CF conceived this work. SY and CF secured the funding. All the authors contributed to the development and refinement of the study protocol. All authors read and approved the final manuscript.

ETHICAL APPROVAL

The ethical considerations of the current study, including the informed consent process and patient privacy measures, are based on the ethics guidelines for medical research in Japan. The study protocol was approved by the ethics committees of the Tokyo Metropolitan University (No. 20083) and the National Center of Neurology and Psychiatry, Japan (A2021‐005). If we amend our protocol, we will ask the ethics committees to review the amendments, and we will report the change in the manuscript.

REGISTRY AND THE REGISTRATION NO. OF THE STUDY/TRIAL

The study protocol was registered in the University Hospital Medical Information Network (UMIN) Clinical Trials Registry (UMIN‐CTR ID, UMIN000044680).

Supplementary Material

Click here for additional data file.