Establishing a Primary Care Performance Measurement Framework for Ontario.

A systematic approach to Primary Care Performance Measurement is needed to provide useful information on a regular basis to inform planning, management and quality improvement at both the practice and system levels. Based on an environmental scan, a summit of primary care stakeholders and a stakeholder survey and supported by Measures and Technical Working Groups, the Ontario Primary Care Performance Measurement Steering Committee, representing 20 stakeholder organizations, identified system- and practice-level measurement priorities and related specific performance measures across nine domains of primary care performance. This initiative addressed measures' selection and technical specification. It did not include data collection. Lessons learned in Ontario can assist other jurisdictions developing frameworks for monitoring and reporting on primary care performance. Cross-country alignment could lead to a coordinated approach to measure and target areas for primary care performance improvement in Canada.

Introduction

A strong primary care system is the backbone of a high-performing health system. For over a decade, Ontario has focused on strengthening primary care delivery. However, the province lacks a coordinated and comprehensive approach to collect, analyze and report data on the performance of primary care at either the practice or system level. At both levels, the paucity of regular feedback on key aspects of performance hinders efforts to identify opportunities for improvement and track the effect of improvement initiatives. At the system level, the meagre information available on primary care performance makes it difficult to monitor and evaluate the effectiveness of policy changes and investments.

In recent years, clinicians, managers and policy makers have increasingly recognized the need for systematic, ongoing feedback on primary care performance. The limited comparative data available on primary care performance at the provincial level – mainly from the Commonwealth Fund International Health Policy Surveys of primary care physicians and the public – indicate that Ontario's primary care performance compares favourably with other provinces, but lags behind international peers, particularly in timely access to care and primary care infrastructure (e.g., primary care teams, electronic medical record [EMR] systems and processes for performance measurement and improvement) (Aggarwal and Hutchison 2012; CIHI 2015, 2016; Hutchison 2013, 2014; Hutchison and Glazier 2013; Hutchison et al. 2011; Marchildon and Hutchison 2016; Osborn et al. 2014, 2015; Strumpf et al. 2012). Individual primary care practices and organizations have had little access to information on their performance, usually restricted to data they collect and analyze themselves. However, most lack the capacity to generate their own performance data.

A number of primary care measurement frameworks have been developed in Canada in recent years, for example, those of Accreditation Canada (n.d.), CIHI (2012), Haggerty and Martin (2005), Haggerty et al. (2007), Hogg et al. (2008), Levesque et al. (2011), Watson et al. (2004). Pan-Canadian and provincial/territorial results for a 16-measure subset of CIHI's 51 primary healthcare indicators have recently been reported (CIHI 2016). Most of the proposed frameworks have been applied on a limited basis, often in a research context. None has been used to report on a province-wide basis on primary care performance at either the organization or system level.

The identification of the need for an overarching framework for strengthening primary care in Ontario can be traced to the 2010 McMaster Health Forum, Supporting Quality Improvement in Primary Health Care in Ontario (Lavis 2010). The Forum participants recommended that a planning group, including representatives of the funder and regulator of health services in Ontario (the Ministry of Health and Long-Term Care [MOHLTC]), and professional associations related to primary care (Ontario Medical Association, Registered Nurses' Association of Ontario, Association of Ontario Health Centres, Ontario College of Family Physicians) develop a strategy for strengthening primary healthcare in Ontario.

In response, the MOHLTC established and chaired the Primary Healthcare Planning Group. The group had a mandate to: (a) draft and build consensus on a strategy for strengthening primary care in Ontario; and (b) plan a meeting where a broad-based group of stakeholders would discuss and finalize the strategy (Ontario MOHLTC 2011). In its final report, the Primary Healthcare Planning Group recommended that “a Working Group be established under the auspices of Health Quality Ontario (HQO) to design a performance measurement framework including indicators to examine how the primary care system is performing against its goals and objectives at the practice, local, regional and provincial levels”.

Beginning in 2012, organizations representing patients and family caregivers, primary care providers, data holders, researchers, managers and policy makers from across Ontario have worked collaboratively to develop a structured approach to measuring primary care performance that can inform decision-making at the practice and system levels. These efforts became the Primary Care Performance Measurement (PCPM) initiative. The initiative was intended as a practical, context-specific exercise to develop an approach to performance measurement that would meet the needs of key primary care stakeholders in Ontario: patients, caregivers, providers, managers, policy makers and the public. It was not designed to develop, refine or validate a conceptual framework for understanding and measuring primary care performance or to establish a generalizable set of performance measures. In this paper, we describe and reflect on the process that Ontario used to develop such an approach to PCPM.

Methods

Establishing the PCPM framework

On November 21, 2012, HQO and the Canadian Institute for Health Information (CIHI), in collaboration with their partners (the MOHLTC, Cancer Care Ontario [CCO], the Institute for Clinical Evaluative Sciences [ICES], eHealth Ontario and Local Health Integration Networks [LHIN]), co-sponsored an Ontario PCPM Summit in Toronto. The Summit was an invitational meeting of senior leaders from key primary care data partners and information users in Ontario. Its purpose was to start laying the foundation for PCPM in Ontario.

To support the Summit, HQO conducted an environmental scan, which examined the current state of PCPM in Ontario, across Canada and internationally. The scan provided a snapshot of existing and recently completed projects that addressed the measurement of performance in primary care settings. The scan included:

A comprehensive literature review, with an electronic search of MEDLINE®, CINAHL, EBSCO Information Services and Google Scholar databases, using the keywords: “performance measurement,” “performance standards,” “conceptual framework,” “outcome and process assessment,” “quality indicators,” “evaluation of primary care” and “design and performance measurement.”

A review of grey literature.

Contacts with organizations throughout Ontario and Canada that HQO knew were doing relevant research or developing performance measurement frameworks for primary care.

The environmental scan identified 19 performance measurement frameworks, initiatives and data sources, and summarized them in a matrix (CIHI 2013; HQO 2013a). HQO used these findings to identify primary care domains that could form the basis for an overarching framework and to select potential measurement priorities (aspects of primary care performance that are valuable to measure at the practice and system level) for each domain. Our goal was not to identify the most comprehensive or rigorous of the existing frameworks, but to select a framework around which potential performance measures could best be organized and presented for consideration by participating stakeholders. Accordingly, we selected HQO's Nine Attributes of a High Performing Health Care System Framework as the most appropriate framework for examining primary care performance in Ontario, in part because the framework was already being used in HQO's public reporting on health system performance and was therefore familiar to many stakeholders. In addition, in its 2011 report, the Primary Healthcare Planning Group recommended it (together with the Triple Aim) as the basis for a primary care measurement framework for Ontario.

The Nine Attributes align with the Triple Aim Framework of the Institute for Health Care Improvement (IHI) and six of the attributes correspond to the Six Aims for Improvement of a Health Care System proposed by the Institute of Medicine (IOM) in its ground-breaking report, Crossing the Quality Chasm: A New Health System for the 21st Century (IHI n.d.; IOM 2001) (Table 1). Because we wished to focus on primary care performance (i.e., outcomes of care and processes linked to outcomes, rather than the structure and organization of primary care), we initially excluded the HQO domain of appropriate resources. However, stakeholders were insistent that it be included on the grounds that primary care performance is contingent on the availability of needed resources.

Table 1.

HQO's Nine Attributes, IOM's Six Aims for Improvement and IHI's Triple Aim

Nine Attributes (HQO)/Six Aims for Improvement (IOM)	Triple Aim (IHI)
	Population health (better health)	Patient experience (better care)	Per capita health cost (better value)
Access*/timeliness§	✕	✕	✕
Integration*		✕	✕
Efficiency*§			✕
Effectiveness*§	✕	✕	✕
Focus on population health*	✕	✕
Safety*§		✕	✕
Patient-centredness*/person-centredness§		✕
Appropriate resources*			✕
Equity*§	✕	✕

HQO = Health Quality Ontario; IHI = Institute for Health Care Improvement; IOM = Institute of Medicine.

HQO.

IOM.

Source: Adapted from Kates et al. (2012).

The Steering Committee for the Summit established the following criteria (weighted equally) to shortlist a set of measurement priorities for the Summit participants to consider:

The information is valuable to have on a regular basis for one or more purposes (e.g., service planning, management or quality improvement) at the practice and/or system (community, regional or provincial) levels.

There is a potential for comparisons of performance across practices, organizations, communities, regions, provinces/territories and/or countries.

The aspect of primary care performance is linked in evidence to one or more components of the IHI's Triple Aim:

Improving the patient experience of care (better care).

Improving population health (better health).

Reducing/controlling the per-capita cost of healthcare (better value).

To facilitate the Summit participants' discussions and priority setting, the Steering Committee prepared and distributed to participants a worksheet of 60 potential measurement priorities and other background materials before the meeting. The committee encouraged participants to share the information and consult widely within their organizations and with their stakeholders.

In total, 61 senior leaders attended the Summit. Following facilitated discussion, they voted for their highest performance measurement priorities, keeping in mind the following question: What aspects of primary care performance would be the most valuable to measure on a regular basis to inform decision-making at the practice and system (community, regional, provincial) levels? The votes were tabulated to generate separate ranked lists of practice- and system-level measurement priorities. The Summit proceedings can be found on CIHI's and HQO's websites (CIHI 2013; HQO 2013b).

Following the Summit, the Steering Committee was broadened to include additional organizations representing primary care providers, hospitals, home and community care providers, patients and family caregivers. In Spring 2013, the organizations represented on the Steering Committee circulated a stakeholder survey to engage their members and solicit their views on the aspects of primary care performance that would be the most valuable to measure. Over 850 people responded. Informed by the Summit and survey results, the Steering Committee finalized the overarching PCPM framework (Figure 1).

Figure 1.

The Primary Care Performance Management framework (Ontario Primary Care Performance Measurement Steering Committee 2014)

Developing specific measures for the PCPM framework

HQO, the Steering Committee, a Measures Working Group and a Technical Working Group collaborated to develop specific measures for the framework (Figure 2). The responsibilities of the three groups were:

Figure 2.

Process to develop specific measures for the Primary Care Performance Measurement framework

Steering Committee: Identify a set of measurement priorities for the PCPM framework based on the Summit and survey results.

Measures Working Group: Select preferred measures for each measurement priority building on and guided by the criteria that were used during the Summit.

Technical Working Group: Advise on technical specifications and infrastructure requirements for data extraction, analysis and reporting.

The Measures Working Group included primary care providers (n = 7), health system decision-makers (n = 5), patients and family caregivers (n = 3), patient relations professionals (n = 2), primary care researcher/clinicians (n = 2) and quality improvement specialists (n = 2). The Technical Working Group comprised primary care researcher/clinicians (n = 3) and data specialists from 10 organizations (n = 15).

Results

Recommended measures

In its Phase One Report, the Steering Committee selected specific measures for the measurement priorities in eight domains of the PCPM framework: access, integration, efficiency, effectiveness, focus on population health, safety, patient-centredness and appropriate resources (HQO 2014). Equity – the ninth domain – was identified as a cross-cutting domain that would be assessed based on a recommended set of 14 economic, demographic and social variables applied to the performance measures in the other domains.

At the system level, 48 (27%) of the 179 recommended measures are currently available at the LHIN level and 90 measures (50%) are available at the provincial level. Available system-level measures are derived from multiple sources, including the MOHLTC's Health Care Experience Survey (HCES), the Canadian Community Health Survey, the Commonwealth Fund International Health Policy Surveys and administrative data sets held by the ICES and CCO. The currency and frequency of reporting vary among the sources.

Only 15 (13%) of the 112 practice-level measures in the PCPM framework will be available in the near future to all primary care clinicians, mainly through the ICES-HQO Primary Care Practice Reports and CCO. Table 2 summarizes practice- and system-level measures by availability.

Table 2.

Summary of specific measures by availability

Domain	Number of measures
Practice and system levels*	Currently reported	Currently reported but modified wording recommended	Not currently available but could be reported using existing infrastructure§	Not currently available but included in survey tool under development; infrastructure required for data collection, analysis and reporting¶	Measures not currently available; new infrastructure required for data collection, analysis and reporting†	Total
Access	8	3	1	0	12	24
Patient-centredness	2	2	12	9	12	37
Integration	15	2	3	1	11	32
Effectiveness	15	1	16	1	43	76
Focus on population health	24	4	4	1	14	47
Efficiency	7	1	8	0	7	23
Safety	3	3	3	0	14	23
Appropriate resources	14	1	0	0	14	29
Total	88	17	47	12	127	291
Equity	Cross-cutting domain – Analysis will be based on 14 population characteristics: age, gender/sex, urban/rural location, ethno-cultural identity, disability, social support, income, education, sexual orientation/identity, language, immigration, aboriginal status, employment status and mental health status

Refers to province-wide (vs. local) availability.

For example, Health Quality Ontario–Institute for Clinical Evaluative Sciences Primary Care Practice Report, Health Care Experience Survey;

Health Quality Ontario Primary Care Patient Experience Survey.

For example, electronic medical record-based measures.

The recommended measures draw on multiple data sources. Administrative data held by the ICES, MOHLTC (Health Analytics Branch) and CCO could provide 23 practice-level measures (21%) and 32 system-level measures (18%). Electronic medical record data could generate 24 practice-level measures (21%) and 13 system-level measures (7%). A practice-level patient experience survey could provide 65 practice-level measures (58%). Population survey data could produce 68 system-level measures (38%). Provider reported that data could provide 47 system-level measures (26%). Finally, primary care organization-reported data could be the basis for 19 system-level measures (11%).

The technical details for each recommended measure, including measure name, description, definition and existing or potential data source, can be found in the Technical Appendices of the Steering Committee's Phase One Report.

Recommendations for implementation

To support the implementation of the PCPM framework, the Steering Committee made a series of recommendations that are outlined in Box 1.

Accelerate efforts to strengthen vendor requirements to incorporate standardized high-value data elements; facilitate standardized data capture, data transfer and exchange; and simplify processes for extracting and analyzing data.

Develop the necessary infrastructure to make the measures available throughout the province at both the practice and system levels, including: (1) a practice-level patient experience survey and a mechanism for pooling EMR data to provide regular feedback to practices over time and allow for comparison with peers; (2) a mechanism for collecting data from primary care providers and organizations; and (3) a mechanism for combining primary care performance measures from multiple sources.

Develop aggregate measures of primary care performance that reflect performance at a broad domain level (e.g., effectiveness), for a more limited area of practice (e.g., management of chronic conditions) or based on a framework such as the IHI's Triple Aim.

Identify organizational responsibility for producing coherent, user-friendly reports using performance measurement data.

Include the PCPM framework measures in new survey tools or updates of existing ones.

Equip primary care providers, organizations, health system managers and policy makers with an understanding of performance measurement, quality improvement methods and leading practices.

Update and revise the PCPM framework, as required, to align with emerging evidence, changing policy priorities, new data sources and evolving information needs, using structures and processes that are inclusive of stakeholders, including patients, caregivers and the public.

Commission an arm's-length formative evaluation of the implementation of the PCPM framework to detect and address implementation challenges and to identify and build on implementation successes.

Recommendations of the Steering Committee to support the transition to better primary care performance measurement

EMR = electronic medical record; IHI = Institute for Health Care Improvement; PCPM = Primary Care Performance Measurement.

As a next step, the Steering Committee committed to selecting two priority subsets of measures and recommended approaches for data collection to support immediate measurement at both the system and practice levels.

Discussion

Lessons learned

Other jurisdictions developing systems for monitoring and reporting on primary care performance can benefit from the lessons learned in Ontario related to stakeholder and patient engagement and resources requirements. Stakeholder engagement has been a key success factor in this initiative. Stakeholder ownership of the recommendations is an important driver of uptake, implementation, sustainability and, ultimately, quality improvements. The collaboration among stakeholders representing patients, providers, data holders, researchers, managers, policy makers and funders from across Ontario has supported alignment and a high degree of consensus on the specific measures that constitute the framework.

Tapping into the experience and expertise of patients and family caregivers through their participation on the Steering Committee and in the Measures Working Group greatly enhanced the meaningfulness and potential value of the recommended PCPM measures. We learned early in the process that the perspectives of primary care providers, managers and policy makers are not always consistent with those of patients and their families. A key to success was the comfort and ability of the patient and family caregiver participants to express themselves freely during the measures selection process. The effect of the partnership with patient and family caregivers is evident from the number of selected measures that are patient-reported – 66 at the system level and 64 at the practice level out of the 199 measures in the framework. In addition, new measures were created and survey-based measures were substantially reworded to reflect the concerns, needs and values of patients and caregivers. Based on this experience, we strongly recommend extensive participation by patients and family caregivers throughout initiatives of this type.

Engagement and consensus-building among a broad range of stakeholders are time-consuming and resource-intensive. The Steering Committee guiding the work met over 20 times, for an average of 4 hours per meeting. The Measures and Technical Working Groups met eight times, each for an average of 4 hours per meeting. Representation included 22 people on the Steering Committee, 21 people on the Measures Working Group and 18 people of the Technical Working Group. The HQO team supporting the work included four dedicated staff and a Senior Advisor who invested, on average, 60% of their time throughout the course of two years with supervision from a senior leadership team. This level of commitment was only possible because the partners in this initiative shared a conviction that a systematic approach to performance measurement and feedback is an essential underpinning for practice- and system-level healthcare improvement.

Not all of the organizations represented on the Steering Committee had collaborated before. However, after working together over a period of two-and-a-half years, the Steering Committee achieved a high level of trust and mutual respect.

The Steering Committee was the final decision-maker in the process with the HQO providing secretariat support. This arrangement helped to build stakeholders' sense of ownership and commitment to implementation of the recommendations.

How could the measures be used?

The recommended PCPM measures were identified through a rigorous and inclusive stake-holder engagement process that was designed to identify performance measures that are valuable for decision-making. Alignment of the PCPM initiative with existing performance measurement and quality-improvement initiatives in Ontario was considered during the framework's development and was facilitated by representation on the Steering Committee and Working Groups of organizations that are leading many of those initiatives.

The recommended set of measures provides a menu of primary care performance measures for use at the system and practice levels. Clinicians, primary care organizations, system managers, policy makers, funders, researchers and organizations representing patients and the public can draw on this bank of recommended measures to meet their diverse needs. The measures can help primary care practices to identify opportunities for improvement and clinicians to evaluate and explore aspects of their performance as part of reflective learning activities. Health system managers, policy makers and funders could use the measures to monitor system performance and the effect of policy initiatives and health system investments. Planners and decision-makers could use the measures to conduct population needs-based planning. Evaluators could use the measures to assess the implementation and effects of innovative primary care programs. Researchers could select outcome measures for use in clinical, health services and policy research in primary care. The measures could help patient-advocacy and civil-society organizations gauge the responsiveness of primary care to the needs and expectations of patients, family caregivers and the public. For some measures, data will be available from existing reports or data sources. In other cases, users would need to collect data themselves.

Data gaps

Our process identified major gaps in data availability at both the practice and system levels. At the system level, important gaps exist for mental health, health promotion, maternal health, family and caregiver experience, comprehensiveness of care and healthcare provider experience. In the short term, data availability could be enhanced by adding additional high-priority measures to the HCES and expanding the number of measures derived from administrative databases held at ICES. At the practice level, key data gaps include mental health and safety. Standardized EMR specifications that facilitate the extraction, pooling, analysis and reporting of EMR data are critically required to improve the availability of high-value practice-level measures.

Relationship to other frameworks

The Ontario PCPM framework differs from other conceptual and measurement frameworks for primary care in a number of respects. Most other frameworks aim to be generic, whereas ours is intended to be specific to the current and short-term-future context of Ontario. Our framework makes a distinction between measures that are applicable at the practice/organizational level and those that are relevant to performance measurement at the system level. Our process was driven by stakeholder perceptions of what would be useful to measure, whereas most other frameworks have been expert-driven. The Ontario framework is populated by a large number and wide diversity of specific measures, reflecting the multiple perspectives represented in the development process. Because this initiative was focused on performance, our framework and specific measures emphasize outcomes of care, processes of care linked in evidence and logic to health outcomes and patients' experience rather than structural and organizational characteristics that figure more prominently in other frameworks.

An initiative is underway in Alberta to develop common quality and outcome measures for the province's Primary Care Networks (PCNs) (Oddie and Krajnak n.d., Scott Oddie personal communication, February 26, 2014). The participants include 25 executive directors, evaluators and provider leads from 13 PCNs and representatives from Alberta Health, Alberta Health Services, Health Quality Council of Alberta, CIHI and academic institutions. Their proposed minimum data set for PCNs includes measures related to access, health promotion, early detection and disease prevention, chronic disease management, patient experience, interdisciplinary teams, cost of services and safety, all of which are included in the PCPM set of measures.

Conclusion

Systematic, regular and relevant performance measurement is essential to identify opportunities for – and the impact of – improvement efforts. A process that is inclusive of key stakeholders can help to ensure the development of a framework that meets the needs of stakeholders and maximizes their commitment to its application to support improvements in policy and practice.

Given the context-specific and stakeholder-driven nature of this initiative, our framework and specific performance measures cannot be generalized to other times and places. However, our methods are transferable and the suite of measures that were selected and technically specified may be a useful source of potential measures for use in other settings. Our process was time- and resource-intensive, in part because of the size and complexity of Ontario's health-care system, and may need to be streamlined in settings where fewer resources are available. If properly aligned, similar processes in other provinces can build a coordinated and sustainable approach to measure primary care performance in Canada. Given the primitive state of routine PCPM in Canada, particular attention needs to be given to the identification of data gaps and the development of processes for collecting and analyzing data to fill those gaps.