Assessing the Perception of Family and Caregivers' Experience with Mental Health and Substance Use Services.

Standardized client feedback surveys encourage a culture of continuous quality improvement, allow for comparison of results over time and across similar types of service providers, and encourage use of evidence-based practices. Recognizing the importance of family and other caregivers in supporting people accessing services for mental health and substance use challenges, a standardized perception-of-care tool (the Ontario Perception of Care Tool for Mental Health and Addictions, OPOC-MHA) was adapted to collect feedback specific to the caregiver experience with these services. A collaborative process engaged a broad range of mental health and/or addiction providers, family advisory networks, and family members and caregivers to identify themes, specific items, and implementation approaches. The final version of the tool evolved through an iterative process of pilot testing and stakeholder feedback. Family member and caregiver perceptions of care will identify service areas in need of improvement, contribute to quality improvement initiatives, and facilitate the comparison of findings over time.

In the health care sector, the measurement of patient satisfaction has a long and storied history (Siegrist, 2013) and now plays a prominent role in performance measurement and quality improvement through its unique ability to incorporate the consumer perspective on both service delivery processes and outcomes. This long-standing interest similarly applies for mental health (Campbell, 1998; Druss et al., 1999) and substance use services (McLellan and Hunkeler, 1998; McLellan et al., 2007), that is, the behavioral health sector more broadly. Research in this sector has focused on theoretical and conceptual contributions (Williams and Wilkinson, 1995), tool development and validation (e.g., Ruggeri, 2010), and the fit with other domains of process and outcome measurement and quality improvement (e.g., Smith et al., 1997; Rush et al., 2014; Carlson and Gabriel, 2001) .

A shift in language occurred over this research trajectory, which involved a movement away from measuring patient or client satisfaction to measuring perception of care. This shift is intended to reflect a move from measuring the reaction to the services received (Graham et al., 1993), from which quality standards are often difficult to interpret, to ask more directly about the care experience in relation to what is expected as standard practice (Einsen, 2010). The nuance is subtle but important, since the range of responses based on a perception of care approach is likely to be wider given respondents may be more willing to report infrequent exposure to, or use of, a practice than express dissatisfaction with this aspect of their care per se. The shift in focus is also consistent with a broader recovery-oriented approach to system design and evaluation that values the perspectives of people with lived and living experience and inclusion in measuring the outcomes of the services they receive (Rose et al., 2011).

The extant body of work on patient and client satisfaction over many decades has focused on the experience of the individual accessing care. Much less attention was given to the experience of family members and other loved ones who are, to say the least, key stakeholders and active participants in the individual’s treatment and support journey (World Health Organization, 2001). In Canada, as in many countries, the importance of family and other caregivers’ participation in the mental health treatment and support of their loved ones is highlighted as a key principle and priority for system reform (e.g., Ministry of Health & Long-Term Care, 2020). This emphasis on the role of the family and other loved ones builds upon a wealth of research evidence about the important role they play in encouraging help-seeking (Urbanoski et al., 2017) and contributing to positive outcomes (Fujino & Okamura, 2009; Innes et al., 2011; Visio et al., 2019) . Other studies have noted that measuring family and caregiver satisfaction with their loved one’s treatment and support may be helpful in improving services (Dourado et al., 2018; Jiang et al., 2019; Resende et al., 2019) . Studies by Gigantesco et al. (2002) and Perreault et al. (2012) show the importance of understanding the collaborative role of family caregivers and how to integrate them into treatment, support planning, and service delivery. The collaboration of families can also strengthen advocacy for the rights of people with mental health challenges, encourage adherence to treatment and support recommendations, contribute to the better management of overload experienced by family caregivers, and improve the relationship between family caregivers and patients (McPherson et al., 2017; Ruggeri, 2010).

Despite these advantages, many challenges exist in involving caregivers in the assessment, planning, and provision of services and supports for the person directly experiencing mental health and/or substance use challenges, with confidentiality being one of the more often cited barriers (Hodgson et al., 2013; Wynaden and Orb, 2005). This highlights the importance of gathering caregiver perspectives to support performance measurement and quality improvement initiatives in addition to the perspectives of clients and patients. Although the experiences of family caregivers and the direct service recipients may differ, they provide complementary perspectives (Garland et al., 2007; Resende et al., 2019). As shown by Garland et al. (2007), when comparing feedback from parents and youth about their experience with a community mental health service, the correlation between youth and parent satisfaction was low. Though youth and parent satisfaction were only marginally related, it is noteworthy that youth race/ethnicity and treatment expectations were predictive of satisfaction among youth, while caregiver strain was predictive of parent satisfaction. As particular predictors of satisfaction were associated with youth or parents, there may be a marked benefit to evaluating service satisfaction differently among these two groups. Similarly, Resende et al. (2019) compared satisfaction across service participants, family members and staff, and found that “feeling supported by the professionals who provide care to the patient” (p. 5) and “receiving information regarding the patient’s psychiatric disorder” (p. 5) stood out as being particularly important to family members. In an inpatient mental health setting, other important factors have been identified, including treatment cost, the amount of direct service received per day, and staffing-related factors, such as perceived adequacy of staff training and bed ratios per patient population (Jiang et al., 2019). Moreover, Missouridou et al. (2021) found that the therapeutic relationship between provider, caregiver, and service recipient is important for treatment efficacy, but providers often feel underprepared to engage families in the treatment process, which may limit the contribution families are able to provide during their loved one’s treatment. Going beyond the traditional inpatient and community-based services, Orlando (2019) conducted a systematic review of patient and caregiver satisfaction with telehealth videoconferences for those living in rural and remote areas. They reported differential factors associated with satisfaction and that, as with other modes of service delivery, the perspectives of caregivers were unique in many respects, especially for those supporting older adults and youth (Orlando et al., 2019). These findings are particularly important given the contemporary impact of COVID-19 on mental health and substance use service delivery and the call for more distance-based and virtual modes of service provision (Vigo et al., 2020).

Recognizing the importance of involving patients and clients in improvement processes for mental health and addictions services, the Ontario Perception of Care Tool for Mental Health and Addictions (OPOC-MHA) was developed, pilot tested, and validated by a research and development team at the Centre for Addiction and Mental Health (CAMH) in Ontario, Canada between 2011 and 2014 (Rush et al., 2014). Key aspects of the new measure were its intended use across both mental health and substance use services, its applicability for both hospital- and community-based organizations, and its focus on measuring perceptions of service delivery that are tangibly amenable to quality improvement and evaluation at the service, organization, or system level. As of June 2020, the OPOC-MHA is implemented in over 230 publicly funded mental health, addiction, and concurrent disorder programs across Ontario, Canada. The aim for a common, standardized measure across this large and diverse service delivery sector was driven by the expressed need for comparative data across like programs, something lacking in the field based on the plethora of tools available at the time in the published or grey literature (Rush et al., 2014).

During the design and validation process of the OPOC-MHA, a version was developed for family and caregivers who themselves were formally enrolled as service recipients in the respective end-user organization. As the new tools were implemented, the need was often expressed for a version designed specifically for family members and other caregivers who are actively engaged in the recovery journey of their loved ones, irrespective of their own formal status as a service recipient. A formal expression of interest for this new version came from the Chair of an advisory network supporting family caregivers who, at the time, was participating on a Provincial Mental Health and Addictions Advisory Council to advise the Ontario government on its new strategic priorities and planning (Ontario’s Mental Health & Addictions Leadership Advisory Council, 2017).

In response to the expressed need for caregiver input on their loved one’s services, the evidence in support of family and caregiver involvement, and the need to minimize barriers to their involvement, a team was re-assembled within the Provincial System Support Program (PSSP) at CAMH to develop a new version of the OPOC-MHA in collaboration with the Ontario Family Caregivers’ Advisory Network (OFCAN). As with the original version of the OPOC-MHA, the decision to develop a caregiver adaptation of the tool was also motivated by the need for a common, scientifically sound instrument that would contribute concretely to quality improvement and sharing of best practices at multiple levels and advance the commitments made to families and other caregivers in government strategic plans in Ontario and more broadly.

The purpose of this paper is to provide a description of the comprehensive tool development and implementation processes for the OPOC-MHA for Caregivers. The following stages of this initiative are outlined: development of the first draft, consultations with stakeholders and end-users, pilot testing, final tool design and development, and implementation plans and processes.

Tool Development Process

Development of First Draft

The initial draft of the instrument built on the extensive foundational development work of the original OPOC-MHA for Registered Clients (Rush et al., 2014). For example, the team aimed, where appropriate, for comparability between versions in key constructs, terms, and even selected items from the original OPOC-MHA. In addition, based on the success of the main version, the goal was to develop a multi-dimensional tool with defined sub-domains of the program experience that still balances completion time and potential respondent burden. This would eventually distinguish the new tool from short, five to 10 item, locally developed tools created in-house at other mental health and addiction organizations that may not generalize to other settings or circumstances (Jiang et al., 2019) and be more in line with the multi-dimensional Brazilian Mental Health Services Family Satisfaction Scale (Bandeira et al., 2002, 2011). The team also aimed for a tool applicable across both mental health and substance use services.

In June 2017, members of the project team met with OFCAN to discuss the feasibility of a new tool that would speak to the priorities of families and caregivers and allow them to provide input into the treatment and support their loved one was receiving. Together they committed to developing a tool that would standardize family and caregiver feedback across the mental health and addiction system in Ontario. The project team began by gathering a sampling of family caregiver feedback surveys currently in use in mental health and addiction programs within Ontario, many represented on OFCAN. These surveys were cross-referenced with one another, and with the registered version of the OPOC-MHA, and common themes were used to identify domains within a new tool—the OPOC-MHA for Caregivers. Through iterative discussions, the following sub-domains emerged for the first draft:

Caregiver involvement and engagement

Caregiver education and support

Caregiver personal support

Environment

Perception of loved one’s care

Overall experience

Individual questions within these domains were identified based on the sample of family caregiver feedback surveys and the OPOC-MHA, with special attention given to (1) the focus and wording of the item to specifically reflect perception of care as opposed to client satisfaction, (2) the usefulness of the responses for quality improvement initiatives, (3) the applicability across multiple treatment contexts within the mental health and addiction sector, and (4) comparability, where appropriate, to the OPOC-MHA.

A total of 38 items emerged from this process, exclusive of demographic items used to categorize and compare responses of sub-groups of family and other caregivers. Selection of the demographic items was deferred to a later stage in the process given the need identified early in the design process for as close a crosswalk with those used in the OPOC-MHA, for comparative purposes.

Stakeholder Consultation

Three focus group sessions and four key informant interviews were conducted with stakeholders in the Ontario mental health and addiction sector. The focus group sessions included one with hospital-based service providers (n = 5 organizations), one with community-based service providers (n = 4 organizations), and one with OFCAN. Participants from the focus group sessions and key informant interviews provided reflections on the applicability and relevance of questions within each domain of the tool, as well as readability of questions and considerations for implementation of the tool. Based on feedback received during the consultation process, adjustments were made to language within the survey to better reflect the perspective of the family member or caregiver. For example, language around education and support was changed to reflect the experiences of caregivers, and descriptions of each domain were added to the next tool version. Consultations also confirmed the importance of including questions that spoke to the process of obtaining consent for caregivers to be involved in the client’s care.

Pilot Testing

A 3-month pilot occurred between June and September 2018, to gather further feedback and learn about early implementation successes and challenges with the new tool in a diverse range of publicly funded mental health and addiction agencies in Ontario. Specific feedback was collected from both caregivers and service providers on the survey questions themselves, as well as the kind of support service providers might require for wide-scale implementation. Feedback from the pilot informed final changes to the tool and assisted the implementation team in developing an implementation strategy specific to the OPOC-MHA for Caregivers.

A group of geographically diverse agencies offering a wide range of services participated as pilot test sites. An OPOC-MHA lead was identified at each site to liaise with the project team throughout the pilot. Site leads were asked to identify the programs at their organization that were interested and appropriate for participation in the pilot and to recruit a total of 20 to 30 caregivers, supporters, or family members of clients at their organization, specifically those caregivers who were not formally enrolled as service recipients themselves. Participating sites were asked to have the survey completed over a 4- to 6-week period. At the end of the pilot testing phase, leads were also asked to share their experiences with the tool, including as it related to language/readability, experience of caregivers, and staff experiences supporting implementation and facilitation of the survey. A short 10-item feedback questionnaire was also administered directly to participating caregivers immediately following their completion of the tool. The questionnaire asked respondents to rate the extent to which questions were clearly worded, easy to understand, and how much the items reflected their experience with the service in question. Although all pilot site organizations had prior experience implementing the standard OPOC-MHA for Registered Clients, site leads also attended a virtual introductory session about the tool and the pilot testing process. They were also informed of support, available throughout the pilot, from an implementation specialist from PSSP at CAMH. The implementation specialist worked with pilot site leads to create an implementation action plan for the caregiver version that complemented, to the extent possible, their implementation approach for the existing OPOC-MHA. Pilot site leads were informed that their own anonymized program-level data would be summarized and returned to them at the end of the pilot phase.

Twelve separate programs across seven agencies agreed to participate in the pilot. At the conclusion of the pilot, nine programs across five agencies submitted completed surveys. Two community-based mental health agencies did not submit completed data; however, one of these agencies participated in the debrief interview at the conclusion of the pilot to discuss potential barriers to implementation or feedback on the survey. As shown in Table 1, the sites represented a cross-section of mental health and addiction agencies in Ontario, including a diversity of program types (hospital, community-based, crisis, supportive housing) and clientele (youth, family, senior populations, inpatient/outpatient).

Table 1

Overview of pilot sites and programs

Pilot site	Overall organizational focus	Program(s)	Number of completed OPOC-MHA for Caregivers surveys
Site 1	Hospital-based mental health	Inpatient Outpatient	21
Site 2	Community-based supportive housing	No separate sub-programs	6
Site 3	Hospital-based mental health	Geriatric Dementia Unit; Dual Diagnosis Unit; Geriatric Transitional Unit	13
Site 4	Community-based crisis services	Crisis services	5
Site 5	Community-based addiction agency for youth and family	Youth Programs Adult Programs	39

Data collection occurred over a 6-week period in June/July 2018. The timing and process of tool administration varied by agency, such as offering the survey at program completion, contacting caregivers via email retrospectively, and conducting a blitz in which caregivers of all clients seen over the 6-week pilot period were offered a survey. The blitz approach allowed agencies to engage participants at different stages of their program or service involvement. One pilot site scheduled time during the last session of a family member and caregiver group program to offer caregivers an opportunity to complete the questionnaire on site. Depending on agency capacity, agencies had the option to offer a paper version of the questionnaire or through a web-based platform (www.surveymonkey.com). Some pilot sites offered both options to participants. Surveys completed via paper copies were entered by agency staff into the Survey Monkey platform at the end of the pilot. Survey respondents were not offered an incentive for participation.

Pilot Site Lead Debrief Interviews

Upon completion of the pilot testing phase, pilot site leads were asked to participate in a 30-min telephone-based interview to share their experience with the questionnaire and their on-site implementation process. Five pilot sites participated in the phone interview, while one pilot site provided written feedback. Debrief interviews occurred over the 2 months following pilot testing. Some site leads involved additional staff in the calls to provide information on staff perspective. Following a semi-structured interview guide, participants were asked questions on the tool itself (e.g., language and readability, length, fit of the questionnaire for their caregiver population) and the implementation process (e.g., how their implementation approach for the caregiver version differed from their use of the OPOC-MHA for Registered Clients, how they introduced and facilitated the surveying process, what worked well with their approach). Sites were also asked about the factors that facilitated or impeded participation rate among caregivers.

As demonstrated in Table 2, the debrief interviews with pilot sites revealed a number of strengths and challenges with the OPOC-MHA for Caregivers. One weakness that was mentioned in the debrief interviews, as well as the stakeholder consults, was the use of the term “loved one.” Tool developers considered multiple terms to replace “loved one,” but consensus was not reached on a new term that may more accurately and broadly capture the caregiver-client relationship. Instead, organizations that use this version of the OPOC-MHA will need to be prepared to answer any questions caregivers may have on the use of the term “loved one.” Multiple sites commented on the inapplicability of several items. However, review of pilot data with one site during the debrief interview revealed that the OPOC-MHA for Caregivers was administered in a program that was ineligible for the survey, as survey respondents were involved in family member programming with the organization. This also resulted in the selection of a high number of “Not Applicable” response options; however, once these responses were removed from the data, there were markedly fewer “Not Applicable” response options selected. This suggests that when implemented with the correct population, the questions were highly applicable to the experiences of caregivers completing the survey. This feedback was useful in determining that implementation strategies need to be improved to facilitate implementation of the tool across the province.

Table 2

Overview of feedback from pilot site leads

Focus of the feedback	Strengths	Challenges
Feedback on the OPOC-MHA for Caregivers tool	• Applicable to caregiver population • Items clear and easy to understand • Appropriate literacy level • Version addresses gap in information that can be used for quality improvement purposes • Appropriate complement to service user (client) feedback • Items align with agency priorities • Themes/domains were comprehensive • Survey format was clear	• Survey was viewed as too long • Demographic questions need to be updated • Use of “loved one” is confusing, especially for those whose relationship to the client may be a service provider • Not available in French
Feedback on the implementation of the OPOC-MHA for Caregivers in pilot agencies	• Interest in continued use at organization • Little facilitation needed • Able to use implementation approach similar to the OPOC-MHA for Registered Clients	• Difficult to implement in crisis and supportive housing settings • Unclear instructions on who is appropriate to complete survey • Lack of motivation to complete survey if no incentive • Difficult to locate or reach family members to offer the survey

N = 6 individuals across 6 pilot sites

Pilot Results

A total of 84 participants responded to the OPOC-MHA for Caregivers questionnaire across the five pilot agencies. See Table 3 for respondent demographic information. The pilot agencies employed a variety of strategies when administering OPOC-MHA for Caregivers questionnaire, such as on paper or electronically via Survey Monkey. Of these 84 responses, 57 participants responded to the 10-item feedback survey about the OPOC-MHA for Caregivers tool. Data from the feedback survey were analyzed using Microsoft Excel. For qualitative data, an open-coding technique was used by study authors to identify either (1) areas of strength or (2) areas of improvement for the OPOC-MHA for Caregivers. The following results include quantitative and qualitative feedback on the survey. When asked to indicate the extent to which they agreed or disagreed that important areas of their experience as a caregiver were covered in the survey, 11 (22.5%) strongly agreed, 31 (63.3%) agreed, five (10.2%) disagreed, and two (4.1%) strongly disagreed. Written feedback on items that were missing from the survey included responses such as two responses indicating that adding an “I Don’t Know” option would be valuable (e.g., “Could have had a category: Not Sure”) for situations in which the caregiver is not present or his or her loved one’s treatment details were kept confidential. Three respondents reported a need to assess difficulties finding the correct treatment type (e.g., “The most difficult aspect is finding the right treatment”; “I did not know where to get help for my son for almost a year”).

Table 3

Respondent demographic characteristics

Number of participants (N = 84)
	n (%)
Age
18–25 years	2 (2.7%)
26–34 years	4 (5.4%)
35–44 years	3 (4.1%)
45–54 years	21 (28.4%)
55–64 years	29 (39.2%)
65 + years	15 (20.3%)
Gender
Female	60 (80.0%)
Male	15 (20.0%)
Race/ethnicity
White	63 (85.1)
First Nations, Inuit, Metis	3 (4.1%)
Asian	4 (5.4%)
Black	2 (2.7%)
Middle Eastern	0 (0.0%)
Latin American	0 (0.0%)
Multiple or Mixed	2 (2.7%)
Relationship to patient/client
Parent	44 (57.9%)
Spouse/partner/significant other	12 (15.8%)
Service provider/peer helper	1 (1.3%)
Sibling	7 (9.2%)
Child	7 (9.2%)
Extended family	1 (1.3%)
Friend	0 (0.0%)
Other	4 (2.3%)

According to respondents, 47 (94.0%) agreed or strongly agreed that the questions in the survey were easy to understand and clearly worded. When asked about which questions respondents found confusing, two indicated that they would have liked explanations for sexual orientation in the demographic section. Another wrote that “the questions were not confusing, but when I chose disagree, I would have liked the opportunity to give an explanation.” Among respondents who had questions about the survey, all received answers or clarification from agency staff facilitating the process. When asked about survey administration, 18 (45.0%) participants responded that they would prefer to complete the survey online, while 22 (55.0%) indicated that they would prefer to complete the survey on paper.

The primary theme that emerged from the qualitative responses from the feedback survey identified concerns with being unaware of or not involved in their loved one’s experience with the service, and, therefore, many of the items seemed not to apply to their situation, furthering the need for an “I Don’t Know” response option for many survey items. Similar to feedback from the pilot site debrief interviews, many respondents stated that multiple questions were not applicable to them as their loved one was not in treatment at the agency at which they were completing the survey, indicating a need to improve the selection process for programs that are appropriate for this tool prior to implementation.

Final Tool Revisions

Based on results from the pilot testing and themes from the feedback survey, the tool underwent a final set of revisions. One significant revision to the questionnaire came from caregiver and agency staff feedback regarding items that may apply to the caregiver’s situation, but which cannot be rated if the caregiver is not involved or is otherwise unaware of that aspect of their loved one’s services (e.g., “You need to add “I don’t know” as an option”). An “I Don’t Know” response option was added for items that the respondent might presumably not know. These items were reorganized into a new domain named “Perceptions of My Loved One’s Care.” Additionally, an “I Don’t Know” response category was added to the residential/inpatient domain as it may be possible that caregivers are uncertain of their loved one’s experience with inpatient/residential treatment services, particularly if they are not present for these aspects of service. All items contain a “Not Applicable” option, and the project team determined that distinguishing between “I Don’t Know” and “Not Applicable” may provide valuable information about caregiver involvement that could be used for quality improvement purposes.

In response to feedback during debrief interviews indicating that the tool was administered to ineligible respondents at one site (e.g., “The survey did not really apply to me because my loved one did not receive treatment here”; “The majority of this survey does not apply to caregivers with loved ones who are not receiving treatment”; “Loved one did not attend this facility”), additional analyses were undertaken to examine the survey results of that program specifically. Since the program offering surveys to ineligible participants completed a high number of surveys, the number of “Not Applicable” responses were highly skewed and then significantly decreased when viewing the data only for eligible respondents. As such, the OPOC-MHA development team made the decision to keep all items despite feedback regarding the inapplicability of some items. See Fig. 1 for a sample item based on all eligible respondents. Finally, in response to pilot site and caregiver feedback, a large portion of the demographic questions were changed for the final version of the OPOC-MHA for Caregivers. The updated demographic questions align with other OPOC-MHA versions and were adapted from the We ask because we care: The Tri-Hospital + TPH health equity data collection research project (Wray et al., 2013), which aimed to develop a consistent, evidence-based approach to collecting socio-demographic information for individuals receiving health care services. The questionnaire still contains demographic items about the caregiver and their loved one.

As there may be variation in mental health and addiction service needs and outcomes according to social and demographic indicators such as gender, age, racialized status, and sexual orientation, the demographic items aim to assess potential inequities in access, quality, and satisfaction of services received by the caregivers themselves or by their loved ones. Therefore, data from the OPOC-MHA for Caregivers can be used to inform and address barriers specific to population groups, to identify quality improvement interventions, and to monitor improvements over time.

Final Version

The final version of the tool contains 41 items, with six items specific to inpatient or residential programming, and a total of 16 demographic items pertaining to both the caregiver completing the survey and the loved one receiving services. The items are categorized into seven quality domains, including caregiver involvement and engagement, caregiver education and support, caregiver personal support, environment, perception of my loved one’s care, overall experience, and residential/inpatient.

OPOC-MHA for Caregivers Implementation

Development of Implementation Material

With funding from and as directed by the Ontario Ministry of the Health and Long-Term Care, the Provincial System Support Program at CAMH provides OPOC-MHA implementation support to mental health and addiction agencies across Ontario. Several resources were developed to aid wide-scale implementation of this new member of the OPOC-MHA “family”—the OPOC-MHA for Caregivers. Specifically, the OPOC-MHA Implementation Guide was adapted to include guidance for the caregiver version, incorporating feedback from pilot sites about their implementation approach and factors that facilitated survey administration. Additional feedback was obtained through a specific consultation session with OFCAN members, who were collaborating partners on the project. Some differences between agencies’ experience with the OPOC-MHA for Registered Clients and the OPOC-MHA for Caregivers were noted; however, multiple agencies stated that their implementation approach between the two surveys was similar. As such, material relevant to the implementation of the OPOC-MHA for Registered Clients is also anticipated to benefit agencies interested in implementing the questionnaire with their caregiver populations.

In the pilot site debrief interviews, site leads were asked about training needs for staff or volunteers with respect to facilitation of this new survey. Many agencies stated that staff and volunteers involved in the pilot testing felt comfortable administering the survey to respondents after reviewing the questionnaire and information letter to caregivers and having team-based discussions with site leads about the implementation process. However, one pilot site lead noted that staff buy-in increased substantially when leadership encouraged survey distribution, signaling the importance of top-down support. In terms of the use of agency staff to administer the OPOC-MHA for Caregivers to respondents, pilot site leads as well as OFCAN members noted the value of utilizing family peer support workers, family navigators, or family resource center staff, when possible.

Consultation with OFCAN members regarding implementation also identified the issue of consent and its impact on surveying a representative sample of caregivers. Members noted that some organizations require the consent of the client to approach his or her family members or caregivers. However, as this survey is about the caregiver’s perception of their loved one’s care, they suggested strongly that consent of the loved one was not necessary to complete the survey.

Finally, information on the caregiver version was added to the recorded OPOC-MHA Orientation Webinar that is viewed by all staff involved in OPOC-MHA implementation at their agencies.

Integration of Questionnaire into OPOC-MHA Database

The Drug and Alcohol Treatment Information System (DATIS) at CAMH provides technical support services and data collection software for community mental health and addiction organizations across Ontario. DATIS developed the web-based version of the standard OPOC-MHA that is hosted and deployed through the DATIS infrastructure at CAMH. The web-based version of the OPOC-MHA for Caregivers survey was integrated into the existing web platform developed by DATIS. Similar to the database administration process for the standard OPOC-MHA for Registered Clients, all data submitted to the OPOC-MHA database for the caregiver version is owned by the agencies but stored by CAMH. As such, CAMH maintains OPOC-MHA for Caregivers data in a separate and secure database within DATIS specific to the OPOC-MHA with the appropriate level of protection to safeguard data confidentiality.

Within the OPOC-MHA database, each in-scope organization and program is coded in a way that can facilitate quality improvement initiatives at the program, organization, or system level. An OPOC-MHA administrator from each agency is able to generate new OPOC-MHA for Caregivers questionnaires, which contain a unique key that links the survey to the appropriate program in the database. The survey key allows respondents to access the website and respond electronically or, if completed on paper, allows agency staff to enter survey results on the OPOC-MHA website through the data entry function. Survey keys are not connected to personal health information, and individual respondents cannot be identified by the survey key in the results. Additionally, the use of survey keys allows agencies to monitor response rates.

The DATIS team developed automated and standardized reports for the OPOC-MHA for Caregivers data. The standardized reports are available from the OPOC-MHA Reporting Portal and accessible to each agency. Agencies also have access to their own raw data, including the open-ended comments following each quality domain. Reports can be filtered in several ways, such as by program, date, and socio-demographic characteristics. Aggregated reports that allow programs to compare their results from collated data from like services across Ontario are also available. Additionally, Ontario’s Ministry of Health and Long-Term Care and its related entities (e.g., Ontario Health regions) have access to all quantitative data on a quarterly basis. The variety of OPOC-MHA for Caregivers reports available provides a unique opportunity for organizations to gather action-oriented, comparable information to inform quality improvement initiatives at the program, organizational, regional, and provincial level.

OPOC-MHA for Caregivers Implementation Support

Each agency in scope for OPOC-MHA implementation support (e.g., any publicly funded mental health or addiction agency in Ontario) works directly with an implementation specialist at PSSP at CAMH as part of the OPOC-MHA team’s approach to support implementation of the family of tools and ensure that providers’ experiential learning is shared provincially. The OPOC-MHA for Caregivers development team provided regular updates to the implementation specialists regarding the caregiver version customization process and worked closely with them to integrate the new version into the support they provide to agencies. This support includes working with agencies to develop an implementation action plan that takes into account the agencies’ caregiver populations while honoring the administration essentials of the tool, such as ensuring completion of the survey is voluntary and providing survey facilitation as needed. Implementation specialists also ensure that each agency’s programs are organized appropriately for reporting purposes, access to the central database has been arranged, and reports can be pulled and filtered. If agencies are interested, support is provided to review and make improvements to their implementation and administration processes.

Discussion

The relationship between client experience and clinical and psychosocial outcomes has been a long-standing interest of researchers and policy makers (Carlson and Gabriel, 2001; Ministry of Health and Long-Term Care, 2020; Visio et al., 2019; Zhang et al., 2018). In recent years, there has been a growing recognition of the importance of family and caregiver involvement in their loved one’s mental health treatment and recovery (Garland et al., 2007; Jiang et al., 2019), although more research is needed to understand the extent to which caregivers’ perception of care can influence both their loved one’s treatment engagement and outcomes. In a study on the challenges of working collaboratively with parents of youth who abuse drugs and alcohol by Misouridou & Papadatou (2017) , mental health professionals were found to lack the skills necessary to engage meaningfully with parents of youth in their care. More research devoted to implementation science within this service sector is needed to understand the barriers to service and system improvement, particularly how to train service providers to engage families during their loved one’s treatment.

Moreover, many jurisdictions have identified that collaboration with families and caregivers is a key principle for mental health system reform and strategy development (Javed & Herrman, 2017; Ministry of Health & Long-Term Care, 2020; Semrau et al., 2016). Unfortunately, many mental health and substance use services are not well-equipped to devote resources to service evaluation and continuous quality improvement. In 2020, a commitment to engage caregivers in mental health and addiction system reform was announced by the Ontario government; however, despite this commitment, there was no standardized family member and caregiver satisfaction or perception of care tool being widely used in Ontario to support service and system improvement. This paper outlines the extensive development and validation process of the OPOC-MHA for Caregivers, a new quality improvement tool that can capture family member and caregiver experience within mental health and addiction services across Ontario. The resulting instrument responds to a need expressed in many other jurisdictions and should be widely implemented with due consideration for evaluation feedback and potential for adaptation for low- to middle-income countries and specific cultural contexts.

With respect cultural and country-specific contexts, there are many aspects of the OPOC-MHA for Caregivers that are universal including the importance of family and caregiver support in the recovery process. That being said, there are a host of contextual factors that may impact implementation of the tool and the responses provided, for example, variation in the availability and accessibility of services, norms around confidentiality of family members’ health information, sensitivity to individual and family cultural needs, and framing of demographic characteristics such as gender and sexual orientation. Further, populations that have been traumatized and/or marginalized such as migrant and refugee populations may find it difficult to offer even constructive criticism of services received for fear of reprisal or loss of these services and supports.

Limitations

This work has several limitations, including the relatively small sample of pilot agencies and limited racial and ethnic diversity of survey respondents. Although the OPOC-MHA for Caregivers underwent a lengthy validation process in Ontario, Canada, these factors may impact its generalizability to other settings.

Conclusions

Through extensive stakeholder involvement and collaboration with provincial mental health and addiction providers, a family advisory network (OFCAN), and family members and caregivers, the OPOC-MHA development team worked to ensure the new tool’s identified themes, items, and implementation approaches met the diverse needs of caregivers involved in mental health and addiction services in Ontario. As a result, the final version of the OPOC-MHA for Caregivers has undergone an extensive process of co-creation through community consultations and pilot testing across geographically and service diverse settings with family member and caregiver respondents. The OPOC-MHA for Caregivers provides organizations offering similar services an opportunity to use a common tool to measure caregiver experience, which encourages a culture of continuous quality improvement, allows the comparison of results, and promotes the sharing of best practices. Organizations can also use their results to inform quality improvement initiatives within a program, an organization, or across the system.

Future work with the OPOC-MHA for Caregivers tool will focus on how the data can be used to contribute to mental health and addiction system enhancement. Research is also needed to establish the relationship between caregivers’ perception of care and client-level outcomes of treatment and support. Additionally, in response to similar feedback about a lack of appropriate quality improvement measurement tools, it is noteworthy that a comparable process has since been undertaken to develop versions of the OPOC-MHA for both supportive housing and crisis services.