System Performance Measurement: Implications for Service Planning.

Improving performance of health and social services is imperative in the face of increasing population needs, complex and chronic patient profiles, and resource scarcity. These three related articles (Performance measurement in mental health and addictions systems: A scoping review; The experience of the treatment demand indicator in Europe: A common monitoring tool across 30 countries; A mixed-methods evaluation of the implementation of a performance measurement system for South Africa’s substance use treatment services) assess numerous system-level efforts undertaken in recent decades and aim at conceptualizing, implementing, and monitoring performance measurement, both in Canada and internationally. Urbanoski & Inglis (2019) report the results of a scoping review on performance measurement in mental health and addiction treatment systems, concluding that a wide variety of indicators is available to those designing a measurement system. Myers et al. (2019) extend this review to cover performance measurement systems for low- and middle-income countries (LMICs) such as South Africa, showing that it is feasible to implement performance measurement systems in LMICs if the system is acceptable, appropriate, and useful. Finally, Montanari et al. (2019) demonstrate the utility of implementing a common treatment demand indicator in Europe, across 30 countries.

These articles suggest that performance indicators permit the follow-up and evaluation of system reforms focused on improving organizational efficiency and interdisciplinary teamwork, while implementing best practices geared toward better population health and patient recovery. Yet consensus is lacking around the meaning of performance: Conceptual frameworks vary according to the programs evaluated (e.g., substance use disorders [SUDs], mental disorders) and across countries. Donabedian’s Quality Framework, a simple and perhaps best-known model, integrates structure, process, and outcomes. Structure encompasses both organizational and provider characteristics and context, whereas process includes interactions between providers and patients regarding treatment. Outcomes relate to the effects of health care for both population and individual health status including patient satisfaction (Donabedian, 1972).

Quality indicators including access, continuity, effectiveness, efficiency, safety, acceptability, appropriateness, and responsiveness (patient-centered care) are key benchmarks in relation to improving population health, with the ultimate aim of reducing SUDs, mental disorders, comorbidity, and suicide rates. System performance indicators draw on population data that is manageable, generalizable, readily available, and affordable to collect. Data used in system performance measurement are usually collected annually and stored in administrative databases at the state level (e.g., the Treatment Demand Indicator [TDI], Montanari et al., 2019). National surveys are other key data sources used in system performance measurement (e.g., the Canadian Community Health Survey 2002 and 2012). One approach merges data from administrative databases with individual patient surveys for improving knowledge on patient care and outcomes.

Use of performance indicators as a monitoring tool for systems, programs, organizations, clinical practices, and patient outcomes is still in its infancy, with fewer available measures for structure than for process and outcomes (Urbanoski & Inglis, 2019). Yet little progress in measuring or monitoring SUD or MH system performance is evident in Quebec (Canada), notwithstanding efforts of the Mental Health Commission of Canada (2015, 2018) and the Commissioner for Health and Wellness (Quebec) (Commissaire à la santé et au bien-être [CSBE], 2012). The database for evaluating SUD programming is similar to TDI in the European context, the “Système d’information clientèle pour les services de réadaptation dépendances: SIC-SRD,” in which data are not centralized at the provincial level and are not necessarily preserved beyond a limited 5-year period. Other important but unresolved quality issues regarding the SIC-SRD involve standardization of data entry and, as with the TDI, the focus on SUD treatment centers offering specialized care to a minority of addiction patients. One attempt was made over several years to monitor performance of MH programs provincially through the “Outil d’alimentation des systèmes d’information sociosanitaires: OASIS,” but it was discontinued because of difficulties related to data collection and management. Medical and hospital databases (RAMQ, Med-Écho), as well as those from emergency rooms (BDCU), public pharmaceutical services, and local community health service centers (I-CLSC) are other sources. However, although centralized at the provincial level (RAMQ), these databases have not been merged, except for the Integrated Surveillance System of Chronic Disease in Quebec (SISMACQ), which is used exclusively by the Quebec National Public Health Institute (INSPQ) and collaborators. Several surveys produced by the Quebec Institut de la statistique, mainly regarding substance use, could also serve as benchmarks for monitoring performance of the SUD program, yet these are not longitudinal surveys and may be dated or not representative of health networks.

Challenges associated with administrative databases in monitoring health system performance, or in service planning, mainly concern issues of responsiveness to needs, validation, and accessibility. Administrative databases provide insufficient coverage of outpatient services and those provided by paramedical professionals in the context of reforms aimed at consolidating primary care and community services. Moreover, little data are collected on patient sociodemographic profiles, and extensive work (e.g., patient follow-up after 30 days of hospital discharge) would be required to obtain data on quality indicators. As administrative databases have been developed for management purposes rather than clinical evaluation, the possibility of missing or inaccurate diagnoses poses validity issues. However, the more administrative databases are used for evaluation, the more they can be improved. As well, wait time for access to Quebec administrative databases is at least 12 months (CSBE, 2017). Other issues concerned noncentralization, fragmentation, variations in data collection in terms of time or space, and the expertise and resources required for analyses of administrative databases for “real time” decision-making. Clinicians and managers also need to be convinced about the relevance of performance benchmarks for improving their practices and for patient outcomes. Additional expertise is required for treating and disseminating data (see Myers et al., 2019). There is considerable room for health system improvement in terms of data collection and utility in supporting the implementation of needs-based planning—essentially “doing more and better with less” for patient recovery!