Performance Measurement in Mental Health and Addictions Systems: A Scoping Review.

OBJECTIVE: The purpose of this study was to evaluate how performance is defined, conceptualized, and measured in mental health and addiction service systems around the world.
METHOD: We conducted a systematic scoping review of English-language scientific and gray literature published from 2005 to 2015. Eligible documents (n = 222) described performance measurement systems and outlined the theory or empirical evidence for indicators. We used a structured approach for data extraction and descriptive and thematic analysis, supplemented with stakeholder consultation.
RESULTS: We identified seven themes in the literature: similarity in performance domains across frameworks; the ability of frameworks to inform care quality at client, program/facility, and system levels; the predominance of indicators of process and outcome, over structure; the lack of evidence on the links between domains and/or indicators; common, but limited, evaluation of family/caregiver involvement; equity as a cross-cutting domain of performance; and limited attention to performance measurement in peer support services.
CONCLUSIONS: The literature on performance measurement in mental health and addictions services is vast, and a wide variety of indicators is available to those designing a measurement system. Evaluations of commonly used performance indicators have yielded mixed evidence on their ability to discriminate high- and low-performing service providers, and their sensitivity to changes in policies and practices. As performance measurement efforts grow in scope and complexity, work will be needed to ensure that indicators are fair, appropriate, and suited to support quality improvement in services of different types.

Performance measurement in health care plays a crucial role in holding the system accountable to maintaining high quality of services for patients and consumers and in helping stakeholders make informed policy decisions (Smith et al., 2008). The primary objective of performance measurement is to monitor and evaluate how well specified measures for performance and quality are met. Population health, individual health outcomes, quality and appropriateness of care, responsiveness of the system, equity, and productivity are common areas of health system performance measurement.

In designing a performance measurement system, stakeholders will follow one of two broad approaches (Loeb, 2004). They may first evaluate what data are available and develop indicators that can be measured using those data. Although this approach is cost-effective (as it uses preexisting data sources), the easiest, cheapest performance measures are often of little value for evaluating quality improvement initiatives and may not adequately represent strategic goals for the system. Alternatively, stakeholders may first determine what should be measured and why, and then design indicators and seek out the required data. Given that health systems involve many and varied stakeholders with diverse needs, performance measurement efforts face a real challenge in serving all of these needs without requiring additional data collection and analysis (Smith et al., 2008).

Performance measurement in the healthcare sector has seen a dramatic increase in recent decades, stemming in part from increased costs and pressures to contain them, as well as patient demands for quality care (Smith, 2005; Smith et al., 2008). The kind of evidence that is generated from performance measurement is valuable for informing system evaluation and enhancement, needs-based planning, and resource allocation. Over and above clinical research and program evaluation that speaks to the effectiveness of individual services and therapeutic approaches, systems research is needed to provide evidence on how well (or not) services are coordinated and delivered so as to affect population mental health and substance-related harms (Babor & Poznyak, 2010; Babor et al., 2008). Despite a growing body of systems research related to substance use, addictions, and mental health services (exemplified in this issue), there is a lack of research that synthesizes evidence across mental health and addictions (MHA) performance measurement initiatives, including systematic evaluation of how high performance is defined, conceptualized, and measured. Shared understandings of system performance may help to guide performance measurement activities, as well as contribute to the development and evolution of service systems themselves.

We addressed this gap in the research literature by conducting a systematic scoping study of MHA performance measurement systems around the world. Our primary aim was to evaluate how performance is defined, conceptualized, and measured. Because the study was conducted to inform the development of an MHA performance measurement framework in the province of Ontario, Canada (estimated population 13.6 million), we also included evidence on frameworks generated for the broader health care system in Canada (in addition to those generated internationally for MHA service systems). This was done to facilitate alignment of the framework development effort with existing initiatives in Canada.

Method

Search strategy

We used the approach described by Arksey and O’Malley (2005) and Levac et al. (2010) to explore how performance has been conceptualized and measured in existing MHA frameworks, including the content of domains and indicators and the purported causal links between them. We included English-language scientific and gray literature published from 2005 to 2015. Documents were identified through keyword searches in online library databases, on websites of key government and academic organizations, and through requests to subject area academic experts (the search strategy, with inclusion and exclusion criteria, is detailed in Box 1). We also searched reference lists of eligible documents.

Scoping review search strategy

Identified documents were screened for eligibility through title and abstract review. Where this was not sufficient to determine eligibility, the full document was reviewed. Eligible documents contained information on framework development, including methods, definitions, concepts, and theories. The primary focus of the study was to review the evidence on performance measurement in MHA service systems internationally. However, as noted above, to be responsive to the needs of local decision makers and health planners, we also searched for evidence on frameworks generated for the broader health care system in Canada. Therefore, documents produced internationally were included if they were focused on performance measurement in MHA service systems, whereas documents produced in Canada were included if they addressed performance measurement in healthcare systems more broadly. There were no other geographical differences in the eligibility criteria used to select documents for review. Ineligible documents focused solely on implementation, ethical issues, stakeholder experiences, statistical issues in risk adjustment, or indicator psychometrics. In cases of report series, the most recent report was selected for inclusion. During the early stages of document screening and selection, 10% of documents were screened by two team members independently to check reliability. Discrepancies were settled by discussion.

Data extraction and analysis

We designed and used a standardized coding tool for data extraction. A random sample of 10% of documents was coded independently by two reviewers, and percentage agreement was calculated for five selected fields. With one exception, agreement across reviewers was reasonably high (above 70%). For the field pertaining to “the associations between domains, between indicators, and/or other correlates and outcomes,” agreement was no better than chance (52%). Discrepancies occurred because of differences across reviewers in judgments as to whether the articles contained this kind of information. Articles were re-reviewed by the project lead (K.U.) for this field.

We used a structured analytical approach to generate a descriptive summary of document types and characteristics, followed by a thematic analysis of data pertaining to conceptualization of performance. For existing frameworks, we tabulated performance domains and population and temporal dimension(s) at which indicators are measured. Population dimensions referred to the level of measurement, categorized as client, program/facility, and system; whereas temporal dimensions referred to structure (the resources needed for service delivery), process (what is done to and for people), and outcome (the results of care; Donabedian, 2005; Gaebel et al., 2015; Rush et al., 2009). For this part of the analysis, we used as a reference the framework for measuring general health system performance in Ontario (Health Quality Ontario, 2014b, 2015b), which is based on the U.S. Institute of Medicine (IOM) framework (Institute of Medicine, 2006). The content of frameworks developed in other jurisdictions was compared to this reference.

Theoretical and empirical findings on the relationships between domains and/or indicators were synthesized separately, with additional themes allowed to emerge to yield a full picture of the conceptualization of MHA service performance. A single article or report could, and typically did, contribute evidence on more than one theme.

Stakeholder consultation

Preliminary findings were shared with stakeholders to obtain feedback and help shape key project messages. Three structured, interactive presentations were conducted in 2016 to a provincial advisory panel convened to guide system enhancement projects in Ontario, Canada (n ∼ 25); a provincial task group on MHA performance measurement (n ∼ 20); and a panel of people with lived experience that meets regularly to contribute to research at a large psychiatric teaching hospital (n ∼ 10). Individual consultations were held with quality improvement specialists working on system enhancement projects in the province (n = 2) and representatives of peer-led/peer-run support organizations (n = 2). The consultations with quality improvement specialists included questions designed to elicit feedback on the results of the thematic analysis (i.e., concerning the utility and feasibility of performance indicators on families and caregivers, equity, peer support, and structure/process/outcomes).The consultations with representatives from peer support organizations focused on discussing the theme of performance measurement in peer support settings. Feedback from stakeholders was synthesized with findings from the literature review, under the relevant themes.

Results

Descriptive summary of documents

The literature search identified 17,510 unduplicated articles and reports (Figure 1). After excluding non–English-language articles, editorials, commentaries, and articles that did not meet the selection criteria, 225 documents remained for review. Of these, 3 theses that were completed at a small Canadian university could not be located. The final review included 222 articles and reports (a full bibliography is available on request).

Figure 1.

Flow chart of document selection

The review included studies conducted in 21 countries across 5 continents (North America, Europe, Australia, Asia, and Africa). The United States contributed the largest number of reports (39.8%), followed by Canada (31.7%), England (5.9%), the Netherlands (4.5%), and Australia (4.5%); the remainder were conducted in various other countries. Studies used a variety of approaches to generate evidence, including focus groups, interviews, expert panels, Delphi process, administrative data analysis, literature review, case studies, surveys, and concept mapping. The majority of reports concerned performance measurement in MHA service systems (80.6%), with the remainder pertaining to general health care services (19.4%). By design, the literature on general health care services was restricted to work conducted in Canada, whereas the literature on MHA services was international in scope. Of the included reports, a minority concerned services for children and youth (10.8%) or substance use and addiction services (18.0%). Most of the literature (69.4% of the 222 reports) concerned performance measurement with in general or multiple health care settings; 13.1% hospital or inpatient services; 7.2% primary care; 6.3% outpatient services (exclusive of primary care); 1.4% services delivered in prisons; 1.4% pharmacy care; and 1.4% emergency or crisis care settings. Document types included original (peer-reviewed) research articles (66.7%), gray literature reports (21.2%), review articles (9.9%), and conference abstracts (2.3%).

Key themes in MHA performance measurement

Results are organized into seven key themes, including the finding of similarity in performance domains across frameworks; the ability of frameworks to inform care quality at client, program/facility, and system levels; the predominance of indicators of process and outcome over structure; the lack of evidence on the links between domains and/or indicators; common, but limited, evaluation of family/caregiver involvement; equity as a cross-cutting domain of performance; and limited attention to performance measurement in peer support services.

Similarity in performance domains across frameworks.

There is a high degree of similarity across frameworks in the ways that MHA performance is conceptualized (Tables 1 and 2). In addition to the six IOM domains, frameworks commonly incorporated measures of appropriateness, integration or continuity, supporting resources (e.g., spending, workforce issues, competence, evaluation), and population health (e.g., prevalence, risk/protective factors, population characteristics). There is a fair amount of conceptual overlap between domains listed separately in the tables. For instance, appropriateness (whether services are relevant to people’s needs and based on accepted standards/evidence) overlaps conceptually with effectiveness. In other frameworks, issues akin to appropriateness/effectiveness were captured through measures of core services specific to a given type of provider (e.g., primary care providers; Barnsley et al., 2005; Hogg et al., 2008; Puszka et al., 2015; Waraich et al., 2010), or to a given care setting (e.g., inpatient psychiatric liaison services; Solomons et al., 2011).

Table 1.

Performance measurement in general health care in Canada

Framework	Performance domains							Population dimensions	Temporal dimensions
	Additional domains	Accessible/timely	Client centered	Effective	Efficient	Safe	Equity	S=System P=Program C=Client	S=Structure P=Process 0=Outcome
Common Quality Agenda (Health Quality Ontario, 2014a, 2014b, 2015a)		✓	✓	✓	✓	✓	✓	SPC	SPO
Health System Performance Measurement Framework (Canadian Institutes for Health Information, 2013, 2015)	• Appropriate resources	✓	✓	(Effective and appropriate)	✓	✓	✓	SPC	SPO
	• Population health
Alberta Quality Matrix for Health (Health Quality Council of Alberta, 2010; Schull et al„ 2011)	• Appropriate	✓	(Acceptable)	✓	✓	✓		SPC	SPO
MOHLTC Health System Strategy Map (Veillard et al., 2010)	• Use of evidence	✓	✓	(Clinical outcomes and health status)	(Productive use and appropriate allocation of resources)	✓		SPC	SPO
	• Availability of data
	• Sustainability and equity
	• Healthy behaviors, health promotion and disease prevention
	• Integration
Hospital Report Framework (described in Yap et al., 2005)	• Internal coordination of care	✓	(Patient satisfaction)	(Outcome)	✓			s	SPO
	• Hospital-community integration
	• Financial viability
	• Liquidity
	• Human resources
Conceptual framework for primary care (Hogg et al., 2008)	• Technical quality of clinical carea	✓	(Comprehensive, Patient-provider relationship)b					p	SP
	• Integration
	• Population health
	• Appropriate resources
McGill University Health Centre Transition Support Office (Biron et al., 2012)	• Continuity	✓	✓	✓		✓		SPC	SPO
	• Work lifec

The core services or clinical activities that fall within the scope of primary care; specifically health promotion and primary prevention, secondary prevention, care of chronic conditions and care of acute conditions.

Comprehensiveness is defined as the “ability to ensure the tailoring of services to health care needs” (Hogg et al., 2008, p. 312), whereas the patient-provider relationship reflects interper sonal communication, respect and trust, whole-person care, cultural sensitivity, family-centered care, and advocacy.

Defined as “the physical, cultural, psychosocial and work/job design conditions that maximize health and well-being of health providers, quality of patient/client outcomes and organizational performance” (Table 1 in Biron et al., 2012).

Table 2.

Performance measurement in mental health and addictions (MHA) service systems in Canada and internationally

	Performance domains							Population dimensions	Temporal dimensions
Framework	Additional domains	Accessible/timely	Client centered	Effective	Efficient	Safe	Equity	S=System P=Program C=Client	S=Structure P=Process 0=Outcome
Canada
Informing the Future (Mental Health Commission of Canada, 2015)a	• Recovery and rights	✓					(Disparities and diversity)	SPC	SPO
	• Promotion and prevention
Addiction System Strategy Map (Ministry of Health and Long- Term Care, 2008)	• Integration	✓	(Quality of Care Matched interventions)b	✓				SPC	SPO
	• System resources & capacity
	• Human resources
	• Sustainability
	• Decision making
Ontario Mental Health System Scorecard (Ministry of Health and Long-Term Care, 2007)	• Standards adherence	✓	(Client and family satisfaction)	✓				s	o
	• Integration
	• Mental health data
	• Capacity
Mental Health and Addictions Quality Initiative (Prince & Willett, 2014)	Client complexity	✓	(Client experience)	(Outcomes)	(Fiscally responsible)	✓		SPC	SPO
	• Employee absenteeism/engagement
Newfoundland and Labrador Centre for Health Information (2015)	• Spending	✓	(Acceptability)	(Quality, Health Outcomes)	✓	✓		SPC	SPO
BC Provincial Quality Indicators (Jones, 2005)	• Appropriate	✓	(Acceptability)	✓	✓	✓		SPC	SPO
	• Continuity
	• Competence
Performance of the addiction and mental health system (Alberta Health Services, 2015)	• Appropriate	✓	(Acceptability)	✓	✓	✓		SPC	PO
Quality of collaborative mental health care (McCusker et al, 2013)	• Comprehensive	✓	(Respectful, Provision of information Responsive, Consumer involvement, Whole-person care)					SPC	SP
	• System coordination
CQI Framework, Fraser Region Child and Youth Mental Health Program (Chovil, 2010)	• Appropriate	✓	✓	✓				SPC	SPO
	• Continuity
	• Competence
Performance measures for Schizophrenia/Early Psychosis Treatment (Addington et al., 2005, 2007, 2009, 2012)	• Appropriate	✓	(Acceptability)	✓	✓	✓		SPC	SPO
	• Continuity
	• Competence
Continuous Enhancement of Quality Measurement in Primary Mental Health Care (Waraich et al, 2010)	• Appropriate	✓	✓	(Outcome)			✓	SPC	PO
	• Early detection
	• Clinical activities for specific conditions and populations
	• Continuity
	• Competence
Mental Health of Children and Youth in Ontario: Baseline Scorecard (Institute for Clinical and Evaluative Sciences, 2015)	• System use	✓		(Outcomes)			✓	SPC	SPO
	• Quality
	• Early identification
	• Children and youth at risk
	• Known prevalence
	• Resources
International
World Health Organization Assessment Instrument for Mental Health Systems/Substance Abuse Instrument for Mapping Services (WHO- AIMS/WHO-SAIMS) (Babor & Poznyak, 2010; Saxena et al, 2007; World Health Organization, 2005)c	• Integration	✓				✓	✓	S	SP
	• Population health
	• Appropriate resources
Quality Index of Service Organization (Costa et al, 2014)	• Programs provided	✓						SPC	SP
	• Coordination
	• Staff supervision
	• Multidisc, teams
	• Evaluation
QQuality measures for international benchmarking of mental health care (Hermann et al, 2006)	• Treatment			(Outcomes)				SPC	PO
	• Coordination
	• Continuity
Australia and New Zealand
National Mental Health Performance Framework (Brown & Pirkis, 2009; National Mental Health Performance Subcommittee, 2013)	• Appropriate	✓	(Responsive)	✓	✓	✓	✓	SPC	SPO
	• Continuity
	• Sustainability
	• Capability
Performance Management Framework, Victorian AOD Sector (Turning Point, 2014a, 2014b)	• Quality	✓	(Acceptable)	✓	✓	✓	✓	SPC	SPO
	• Appropriate
	• Continuity
	• Sustainability
	• Competence
Key Performance Indicator (KPI) Framework for New Zealand Mental Health and Addiction Services (New Zealand Mental Health and Addictions KPI Programme, 2010, 2014, 2015)	• Appropriate	✓	(Responsive)	✓	✓	✓	✓	SPC	SPO
	• Continuity
	• Sustainability
	• Capability
Crisis Reliability Indicators Supporting Emergency Services (CRISIS) framework (Balfour et al, 2016)	• Least restrictive	✓	✓	✓		✓
	• Partnership
Europe
Performance Indicators for public mental health care, Netherlands (Lauriks et al, 2014)	• Appropriate	✓		✓				SPC	PO
	• Continuity
Mental Health Benchmarking Project, Scotland (Coia & Glassborow, 2009)	• Sustainability		(Patient experience)	(Health outcomes)	✓			SPC	SPO
Performance indicators for mental health services, Ireland (Carrick et al, 2013)	• Appropriate	✓	(Acceptable)	✓				SPC	SPO
	• Coordination
	• Population characteristics
	• Appropriate resources
UKUK Association of Public Health Observatories report on mental health (Wilkinson et al, 2008)	• Interventions		(Service user experience)					SPC	SPO
	• Effectiveness of partnerships
	• Risk/protective factors
	• Population health status
	• Workforce capacity
Africa
Performance measurement framework for substance abuse treatment, South Africa (Myers et al., 2015)	• Qualityd	✓	✓	✓	✓			SPC	PO
United States
IOM (Institute of Medicine, 2006; Pincus et al., 2007; Pincus et al., 2011)		✓	✓	✓	✓	✓	✓	SPC	SPO
National Behavioral Health Quality Framework (SAMHSA; http://www.samhsa.gov/data/national-behavioral-health-quality-framework; Patel et al., 2015)	• Healthy livinge	✓	✓	✓		✓		SPC	PO
	• Coordination
Washington Circle (Garnick et al., 2006, 2012)	• Continuity	✓						SPC	p
Quality indicators for outpatient child mental health care in California (Zima et al., 2005)	Completeness of the clinical assessment					✓		SPC	SPO
	• Following basic treatment principles
	• Appropriate psychosocial treatment
	• Medication referrals
	• Linkage to other service sectors
Quality of Care Framework for child mental health care (Vargo et al., 2013)	• Appropriate	✓	(Consumer engagement)	(Outcomes)				SPC	SPO

These domains correspond to strategic directions put forth by the Mental Health Commission of Canada.

Defined as consideration for clients’ values, preferences and expressed needs in the care they receive.

The WHO-AIMS/WHO-SAIMS are tools for assessing regional or national mental health care and substance use systems, respectively. Items are grouped into subscales assessing the jurisdiction’s policy and legislative framework, population need for services, the types and characteristics of specialized and primary MHA services, human resources, public education and links with other sectors, and monitoring and research. Items were cross-referenced with the performance domains in the CQA.

Measured in terms of client engagement and perceptions of care and therefore aligned closely with acceptability or client centeredness.

Defined as assistance to communities in using evidence-based activities to foster healthy living, such as the use of prevention services; screening and assessment for alcohol, tobacco, and other drugs; housing; legal system involvement; and medication side effects.

These basic findings with respect to performance domains were reflected in frameworks that emphasized strategic goals during development (e.g., Mental Health Commission of Canada, 2015; Ministry of Health and Long-Term Care 2008; Veillard et al., 2010), as well as those that used techniques such as concept mapping or other group-based activities to identify the main components of performance (Graham et al., 2014; Holmes et al., 2014; McCusker et al., 2013; Nabitz et al., 2005a, 2005b; Resnick & Griffiths, 2010; Roeg et al., 2005; Sayal et al., 2012; Vargo et al., 2013). For example, in a study of performance definitions across stakeholder groups (clinicians, health system managers, patients, policy makers, and researchers), accessibility, continuity, patient centeredness, comprehensiveness, coordination, effectiveness, equity, and safety emerged as key components, named by three or more of the five groups (UBC Centre for Health Services and Policy Research, 2015). Frameworks consisting of lists of indicators, without the organizing structure provided by domains, also did not contribute any new information over and above the more structured frameworks (e.g., American Society of Addiction Medicine, 2014; British Columbia Office of the Provincial Health Officer, 2015; Cole et al., 2012; Holmes et al., 2014; Laugharne & Shankar, 2009; Levitt et al., 2014; Nigam et al., 2008; Parameswaran et al., 2015; Samu et al., 2011).

In our own stakeholder consultation, discussions with people with lived experience of the system highlighted the importance of responsiveness (whether services are responsive to the needs, expectations, and preferences of service users and their families). Stakeholders also noted a preference for the term person-directed care, over client- or person-centered care.

Ability to inform care quality at client, program/facility, and system levels.

With a couple of exceptions, all frameworks had the capacity to offer insights into care quality at the client, program or facility, and system levels (second to last column in Tables 1 and 2). Frameworks contained both indicators specific to a given level of measurement (e.g., provider density is inherently a system-level measure), and indicators that were defined at a lower level but could be reasonably rolled up for interpretation at a higher level (e.g., wait times are measured for individuals who use services, but also provide information at program and system levels).

Overall, few framework development initiatives addressed the issue of population dimensions explicitly (see Mental Health Transformation Workgroup, 2011; Turning Point, 2014b, for exceptions). During framework development in the state of Victoria, Australia, service providers identified client-level measures as the most relevant to quality improvement in their services (measures of wait times, follow-up activities, reductions in substance use, and client satisfaction), whereas accreditation (a program-level measure) was the indicator to which they most felt they could be held accountable (Turning Point, 2014a). Links between services (a system-level measure) were also seen as both useful and important for accountability.

Predominance of indicators of process and outcome, over structure.

Much greater attention was paid in the literature to temporal dimensions, likely owing to the traction of Donabedian’s conceptual framework for assessing quality in health care (Donabedian, 2005). A number of reports framed their evaluation of performance solely or primarily in terms of structures, processes, and outcomes (e.g., Cheng et al., 2010; Dausey et al., 2009; Grabowski et al., 2010; Kilbourne et al., 2010; Meehan et al., 2007; Roeg et al., 2005; Schaub et al., 2013). Most frameworks contained elements of all three dimensions (last column in Tables 1 and 2).

Structure tended to be the least frequently measured temporal dimension, typically incorporated as measures of sustainability or appropriate resources. In some cases, structure was limited to single measures of provider competence, or whether evaluations are conducted (Addington et al., 2005, 2007, 2012; Chovil, 2010; Waraich et al., 2010). A number of reports cited a need for greater attention to exploring and developing structural indicators (Baars et al., 2010; Hogg et al., 2008), whereas others have highlighted specific structural features as key priorities for future indicator development work (Canadian Institutes for Health Information, 2015; Garnick et al., 2012). In a study focused on conceptualizing structure in addiction treatment settings, aspects identified by service providers included inter-organizational cooperation and coordination, finances and facility characteristics, professionalization, staff autonomy, and job requirements (Roeg et al., 2008).

Process measures predominate in performance measurement (Henderson et al., 2014; Lauriks et al., 2012; Patel et al., 2015). In addition to being relatively easy to capture with existing administrative data, there is a clear link between treatment process and provider accountability and costs (Asch et al., 2011; Carrick et al., 2013; McLellan et al., 2007; O’Brien et al., 2007). Relative to outcomes, processes are also more actionable and sensitive to differences in performance (Baars et al., 2010; Watkins et al., 2010). An example of development work that has focused heavily on treatment processes is given by the Washington Circle, a U.S.-based multidisciplinary expert panel convened to develop performance measures for addiction treatment (Garnick et al., 2006, 2009, 2012).

Outcome measures were also typically present in some form. The most common outcomes captured in existing frameworks related to service use (e.g., retention, length of stay, 30-day readmission), symptoms and functioning, and client or family perceptions of care.These are consistent with outcome domains identified as priorities by administrators and health planners (Herbeck et al., 2010). Less commonly measured outcome domains were quality of life, substance use, or social relationships (a full analysis of outcome domains tabulated across frameworks is available on request).

There was variability in the ways that outcomes are monitored. The importance of capturing multiple perspectives (e.g., client, family, provider) to get the full picture of service quality was noted (Gaebel et al., 2015; Institute of Medicine, 2006; Lauriks et al., 2012; National Mental Health Performance Subcommittee, 2013). On point, client perceptions of care, a common outcome measure, tend to vary more within than between providers (reviewed in Ruud, 2009). It is entirely legitimate for clients to experience care delivered by a single provider differently; however, this finding does raise questions as to whether client perceptions can be used to distinguish between high- and low-performing providers (this is true even when the clients’ reports of their own experiences is valid). Several jurisdictions have opted for routine outcome monitoring using standardized tools, such as Health of the Nation Outcome Scales (HoNOS; Alberta Health Services, 2015; National Mental Health Performance Subcommittee, 2013; New Zealand Mental Health and Addictions KPI Programme, 2010). In addiction service settings, routine monitoring has traditionally been limited to urinalysis and measures of treatment attendance, although more recent efforts to monitor progress and outcomes over the course of treatment have emerged (Goodman et al., 2013; McLellan et al., 2007).

Lack of evidence on the links between domains and/or indicators.

Conceptual linkages between domains are rarely specified in performance frameworks (see Canadian Institutes for Health Information, 2015, for an exception). Relative to linkages between domains, the associations between indicators are more commonly addressed, although it is still the case that the causal relationships between indicators are typically either not specified or not validated through research (Lauriks et al., 2012). Demonstrating empirical links between indicators, particularly across temporal dimensions (i.e., structures, processes, and outcomes), gives evidence of indicator validity. There is evidence linking national, regional, and facility characteristics (e.g., income inequality, gross domestic product, material deprivation, provider density, facility size) with care processes (Costa et al., 2014; Olfson et al., 2010), but not necessarily outcomes (Desai et al., 2005; Hendryx, 2008). A substantially larger body of literature has examined the links between measures of processes and outcomes; for instance, showing better outcomes among clients who meet benchmarks for care continuity and the frequency of visits in the initial weeks of treatment (Desai et al., 2005; Garner et al., 2010; Garnick et al., 2007; Greenberg & Rosenheck, 2005; Harris et al., 2010; Rost et al., 2005). Although documented at the client level, many of these associations have not been replicated with indicators aggregated to the facility level (Desai et al., 2005; Harris et al., 2007, 2009), raising questions about the ability of these indicators to discriminate between high- and low-performing facilities.

Common, but limited, evaluation of family/caregiver involvement.

Family-centered care, family–provider relationship, and, to a lesser extent, provision of direct services to family are often incorporated as features of effectiveness and person-centeredness or responsiveness (e.g., Addington et al., 2012; Balfour et al., 2016; Chovil, 2010; Hogg et al., 2008; Jones, 2005; New Zealand Mental Health and Addictions KPI Programme, 2010; Vargo et al., 2013; Waraich et al., 2010; Zima et al., 2005). Family involvement in care was particularly common in frameworks for youth (Chovil, 2010; Vargo et al., 2013; Zima et al., 2005). Australia’s National Mental Health Performance Framework offered the opportunity for potentially more comprehensive representation of family and caregivers by specifying both client and caregiver in their definitions of service effectiveness, appropriateness, responsiveness, and safety; however, indicators do not appear to have been developed in all of these domains (Brown & Pirkis, 2009; National Mental Health Performance Subcommittee, 2013).

Equity as a cross-cutting domain.

Equity is typically conceptualized as cross-cutting other performance domains, allowing for exploration of how features of effectiveness, accessibility, safety, and so on are distributed across the population. Indicators are broken down by diagnosis and various sociodemographic characteristics (e.g., age, sex, gender identity, ethno-cultural background, immigrant status, socioeconomic status). Others have noted the importance of moving beyond patient characteristics to incorporating measures of program and system structures in the evaluation of equity (e.g., the numbers and characteristics of people who are excluded from services because of admission rules or capacity issues; the sociodemographic characteristics of staff members; staff training and support for culturally safe practice; the use of interpreters) (Nakaima et al., 2013; Wong et al., 2011, 2014). In Australia’s system of MHA services, equity is used in framework development, with applicability across diverse populations being one of the criteria for indicator selection (National Mental Health Performance Subcommittee, 2013).

Limited attention to performance measures in peer support services.

Very limited attention is given to peer support services in the performance literature, despite their established presence in many jurisdictions. Most commonly, where peer support was reflected in existing frameworks, it was as a single indicator focused on availability, with no information captured on capacity, scope of activities, or effectiveness (Ganju et al., 2005; Mental Health Transformation Workgroup, 2011; Oregon Health Authority, 2014; Parameswaran et al., 2015). It was also unclear whether frameworks included peer support service within their scope (i.e., whether they were expected to report on and be held accountable to the same performance indicators as other services).

We identified one initiative to develop quality indicators specifically for parent-delivered support services in child mental health (Kutash et al., 2014; Olin et al., 2014a, 2014b). This initiative underscored some of the challenges inherent in applying traditional performance measures, with authors citing expert disagreement over whether the use of standardized assessment protocols was consistent with the principles of peer support. We heard similar feedback in our stakeholder consultation, with stakeholders from peer support organizations noting that the collection of unique identifiers for members is inconsistent with the values of peer supports and could be expected to interfere with recovery.

Despite the challenges, it was a repeated theme, in both the literature and our stakeholder consultation with peer support organizations, that some form of performance measurement is crucial to establishing the legitimacy and ensuring the viability of authentic peer support services going forward. The development of customized indicators that align with the values and objectives of peer support services was seen as both possible and preferable to exclusion from such initiatives.

Discussion

Our primary objective was to conduct a comprehensive and systematic conceptual exploration of performance in MHA service systems. Using a structured process, we identified a large number of documents from the academic and gray literature (222), which was supplemented with consultations with a variety of stakeholder groups. Our synthesis of findings highlights a number of key considerations for jurisdictions engaged in designing and implementing performance measurement activities in the MHA service sector.

There are a finite number of ways to measure performance, and so there were, on the whole, few notable differences between frameworks in the content of performance domains and indicators. The majority of existing MHA performance measurement frameworks balance measures that can be used to express information about clients, programs/facilities, and systems (i.e., they yield information about variability across episodes of care and service providers, as well as speaking to system and population health). From a public health perspective, it is of interest to understand the cumulative impact of a service system on population health (e.g., incidence of substance-related harms, substance use disorders, and suicide, crime rates, HIV infection; Babor et al., 2008). Overall, population health indicators featured in only a minority of frameworks, and there was little attention overall to the potential for performance measurement initiatives to further our understanding of the population health impacts of treatment and other supports. This is an area for future research, particularly with respect to substance use and addictions, as stakeholders develop strategies and policies to respond to escalating rates of substance-related harms in their jurisdictions (Jones et al., 2018; Wood, 2018).

Measures of process (what is done to and for people) and outcomes (the results of care) are relatively more common than are measures of structure (the resources that are needed for service delivery). There are a number of likely reasons for this. In addition to being difficult to define, a favorable structural measure indicates only the capacity for, rather than the occurrence of, high-quality services (Henderson et al., 2014)—it indicates only whether the conditions were in place for good quality care. That said, challenges related to chronic underfunding, including infrastructure shortcomings and difficulties recruiting and retaining staff (Eby et al., 2010; Gallon et al., 2003; McLellan et al., 2003; Ogborne & Graves, 2005) may be a rationale for measuring and monitoring structural inputs specifically in the MHA service sector. To the extent that they lend a perceived sense of fairness to performance measurement (for instance, by allowing service providers to highlight their challenges in meeting treatment demand and delivering high-quality services), they may be important tools for advocacy and equity in policy development. Furthermore, measuring outcomes in the absence of client and population characteristics runs the risk of rewarding good outcomes without controlling for illness severity or social determinants of health (Baars et al., 2010)—suggesting that the robust measurement of outcomes requires measures of structure. Despite recognized challenges, developing valid measures of structure was a stated priority for future work in many jurisdictions (Canadian Institutes for Health Information, 2015; Garnick et al., 2012).

It was noted that the causal relationships between indicators are typically either not specified or not validated through studies (Lauriks et al., 2012), but that the relationships between health system performance and actual health outcomes are varied and not always in the expected direction (Arah & Westert, 2005). This gap in evidence hinders our ability to use performance data to derive policy that will maximize population health (Etches et al., 2006). Evaluations of commonly used performance indicators (e.g., continuity of care, service intensity, client perceptions of care) have yielded mixed evidence on their ability to discriminate high- and low-performing service providers, and their sensitivity to changes in policies and practices. Measures that vary more within than between providers, as has been found for client perceptions of care, may not be suited to performance measurement if the purpose is to distinguish between high- and low-performing providers within the system. However, such findings also highlight the need to look at whether and how performance measures vary across client subgroups (e.g., whether women or people with a particular diagnosis tend to have a better or poorer experience with a given provider). In evaluations of the Washington Circle performance measures, the positive association between treatment engagement (based on service frequency in the initial weeks of treatment) and arrest was found to be weaker in Black and Latino clients (Acevedo et al., 2015), whereas its association with hospital readmission was apparent only among those who had a prior suicide attempt (Glass et al., 2010). Much more work of this nature is needed to clarify the interpretation and the use of findings from performance measurement in service planning and policy development.

There is evidence of a potentially widening role for equity in performance measurement (e.g., Nakaima et al., 2013; Wong et al., 2011, 2014), with measures moving beyond patient sociodemographic characteristics to structural inputs. Although sex and gender figure prominently within existing frameworks, we identified only one framework that included parental status in the evaluation of equity (Turning Point, 2014b). Given the gendered nature of barriers to MHA treatment, particularly for women who are pregnant and parenting (Cormier et al., 2004; Greenfield et al., 2007), monitoring service access and effectiveness by parental status is a notable gap in existing frameworks.

Finally, there are legitimate challenges associated with implementing performance measurement in peer support services, combined with a general lack of examples in the literature on ways in which this has been done successfully in the past. As performance measurement in MHA continues to evolve in Canada and elsewhere, efforts should be made to partner with peer support organizations to ensure both that the services they offer are represented in these strategic initiatives and that the indicators on which they are evaluated are fair, appropriate, and able to support quality improvement.

The vast majority of evidence that we reviewed was produced in high-income countries, generated in relatively well resourced service systems. In this respect, our study was likely limited by the exclusion of non–English-language documents and biased toward representing MHA systems in Western countries. We identified only one document from South East Asia (Singapore) and one from Africa (South Africa). No documents were identified from India, China, or South America (however, it should be noted that some of these countries were represented in the international studies that we reviewed). Systems research and performance measurement are less common in low- and middle-income countries (LMIC; Babor, 2017; Myers et al., 2012, 2015), where infrastructures for data and other supports may be lacking (Myers et al., 2014). Efforts to establish international quality indicators for substance use treatment that include LMIC (United Nations Office on Drugs and Crime/World Health Organization, 2016) may help to expand performance measurement initiatives globally.

Within this context, the present study provides a valuable empirical synthesis of how the concept of performance has been operationalized in MHA systems, albeit in predominantly high-income countries. The literature on this topic is vast. Systematic and structured syntheses such as this one are valuable tools for highlighting key considerations for researchers and planning bodies embarking on performance measurement activities in the MHA service sector. Offering a critical reflection on the state of performance measurement and definitions of care quality, findings may likewise serve to guide the development and evolution of MHA service systems, whether they are being newly established or expanded and enhanced to better meet population need.

As described, although precautions were taken to promote inter-coder reliability in study selection and coding, low reliability was obtained for one key field related to the coding of information on associations between domains and/or indicators. We attempted to minimize the impact of this limitation through re-coding the field.

Despite the limitations, we are confident that this study offers a reasonably robust picture of how performance has been conceptualized in MHA service systems globally. Future key considerations for performance measurement activities include the implementation of indicators to better capture the population health impact of MHA service systems and help drive public health responses to reducing substance-related harms, as well as the development and implementation of structural indicators to better capture system contexts and resource needs. Given the significant role played by peer support in MHA services, additional work is also needed to establish how its role can best be captured within performance measurement systems.

Finally, greater attention is needed overall to the causal linkages between indicators. This is particularly crucial to avoiding the creation of performance measurement systems that simply rely on available data, rather than capturing accepted and understood elements of care quality. Evaluations of commonly used performance indicators have yielded mixed evidence on their ability to discriminate high- and low-performing service providers, and their sensitivity to changes in policies and practices. As performance measurement efforts grow in scope and complexity, work will be needed to ensure that indicators are fair, appropriate, and suited to support quality improvement in services of different types.

Acknowledgments

Karen Urbanoski holds the Tier 2 Canada Research Chair in Substance Use, Addictions, and Health Services Research from the Canadian Institutes for Health Research. The authors acknowledge Alex Abramovich and Miroslav Miskovic for research support in selecting and coding articles and Betty-Lou Kristy and Deborrah Sherman for supporting the development of the theme on peer support services. The authors also thank all stakeholders who provided feedback during the course of the project.

Databases/search engines: PubMed, MEDLINE, Cochrane, EMBASE, CINAHL, PsycINFO
Terms (used in combination): Performance measurement, performance indicators, performance frameworks, quality indicators, quality improvement, system performance, scorecards, report cards; quality care, quality reports, healthcare quality, health system quality; health system evaluation; health, healthcare, medical care; mental health, mental healthcare, psychiatry, substance abuse treatment, addiction treatment
Organizational websites (searched by hand): World Health Organization (WHO); Organization for Economic Cooperation and Development (OECD); Canadian Institute for Health Information (CIHI); Health Canada; Canadian Centre on Substance Abuse (CCSA); Institute for Clinical and Evaluative Sciences (ICES; Canada); Ministry of Health and Long-term Care (MOHLTC; Canada); Health Quality Ontario (HQO; Canada); Centre for Health Services and Policy Research (Canada); US Department of Health and Human Services; U.S. Department of Veterans Affairs; King’s Fund (UK); Centre for Reviews and Dissemination (CRD; UK); National Institute for Clinical Excellence (NICE; UK)
Inclusion criteria: Published between 2005 and 2015; English language; peer-reviewed studies, including primary studies (original research) and reviews; gray literature, including government and evaluation reports; documents that outline MHA performance measurement frameworks and their development; documents that outline frameworks for general health care and health system performance measurement in Canada (e.g., hospital care, primary care); literature that describes the prevalence, correlates, and outcomes of existing performance indicators
Exclusion criteria: Commentaries, editorials; documents on performance measurement in areas other than health; documents that describe the purpose and goals of performance measurement without reference to performance domains or framework development; documents that describe statistical or methodological issues (e.g., risk adjustment, validity/reliability); documents that describe stakeholder views, experiences, ethical issues, or implementation of performance measurement (without describing the framework and its development); descriptions or evaluations of educational initiatives for providers or clinical guidelines; documents that outline performance measurement frameworks in Canada that are specific to nonMHA conditions (e.g., heart disease, cancer, diabetes); program evaluations, evaluations of specific services, types of interventions, or innovations in practice; evaluations of quality improvement initiatives that do not involve performance measurement; original research or reviews that report on treatment processes and outcomes generally, but not within the scope of a performance measurement strategy or framework