Safety-net institutions in the US grapple with new cholesterol treatment guidelines: a qualitative analysis from the PHoENIX Network.

BACKGROUND: Clinical performance measures, such as for cholesterol control targets, have played an integral role in assessing the value of care and translating evidence into clinical practice. New guidelines often require development of corresponding performance metrics and systems changes that can be especially challenging in safety-net health care institutions. Understanding how public health care institutions respond to changing practice guidelines may be critical to informing how we adopt evolving evidence in clinical settings that care for the most vulnerable populations.
METHODS: We conducted six focus groups with representatives of California's 21 public hospital systems to examine their reactions to the recent 2013 cholesterol treatment guideline.
RESULTS: Participants reported a sense of confusion and lack of direction in implementing the new guideline. They cited organizational and data infrastructural inadequacies that made implementation of the new guidelines impractical in their clinical settings.
CONCLUSION: Adopting new performance measures to align with evolving cholesterol guidelines is a complex process that may work at odds with existing quality improvement priorities. Current efforts to translate evidence into practice may rely too much on performance measures and not enough on building capacity or support for innovative efforts to meet the goals of guidelines.

Introduction

Translating evidence into routine clinical practice is critical to providing high-quality care for individual patients and improving health at the population level. Practice guidelines are, in theory, syntheses of best available evidence toward integrating the most current scientific evidence in clinical practice, but implementation of guidelines in clinical care remains suboptimal.1,2

New guidelines and updates often require development of corresponding performance metrics and concerted centralized quality improvement (QI) efforts to adopt new evidence in clinical practice. Such system changes can be especially challenging in safety-net health care institutions that take care of vulnerable populations at highest risk for poor outcomes. These safety-net health systems are less likely to have adequate health information technology systems, and clinicians at these institutions face significant barriers (eg, lack of dedicated time) to engagement in QI.3–6 Understanding how the public hospitals and safety-net health systems respond to new guidelines may be critical to informing future guideline development and adoption of evolving scientific evidence in clinical settings that care for high-risk populations.

The 2013 American College of Cardiology/American Heart Association (ACC/AHA) guideline for treatment of blood cholesterol7 represents an opportunity to examine how resource-limited, safety-net health systems that are required to report quality measures respond when guidelines are changed. In 2013, the ACC and AHA published a new cholesterol guideline replacing the clinical model of pursuing prespecified low-density lipoprotein cholesterol (LDL-C) targets, with one that has an emphasis on matching the intensity of medication treatment (ie, statin) with cardiovascular risk (Table 1).7 Though it met with mixed reviews,8–10 the new cholesterol guideline has changed the current standards for both clinical practice and performance measurement in cholesterol management. The experience of California’s public hospital systems in implementing and sustaining QI activities for cholesterol management in the setting of changing treatment guidelines offers a unique opportunity to examine how health care leaders and QI professionals on the ground responded to the new ACC/AHA cholesterol guideline in public hospitals and safety-net health systems. Findings could have important implications for health policy, both for cardiovascular disease prevention as well as for translating novel guidelines into clinical practice.

Table 1

Key comparisons of the 2013 ACC/AHA cholesterol guideline with the previous cholesterol guideline (ATPIII)

	ATPIII Guidelines25–28	2013 ACC/AHA Guideline on the Treatment of Blood Cholesterol7,25,29
Risk estimation	Estimated 10-year risk of coronary heart disease based on the Framingham risk score (FRS) equation	Use of new Polled Cohort Risk Calculator to estimate 10-year risk of atherosclerotic cardiovascular disease (ASCVD)
Target groups	Focused on patients with multiple risk factorsInitiation of lipid-lowering therapy recommended at the following LDL-C levels:Risk category High risk, FRS ≥20%: ≥100 mg/dL Moderately high risk, FRS=10–20%: ≥130 mg/dL Moderate risk, ≥2 risk factors, FRS<10%: ≥160 mg/dL Low risk, 0–1 risk factors: 190 mg/dL	Identified individuals 21 y or older who fall into any of the following 4 risk groups as benefit groups for statin therapy:• Individuals with known ASCVD• Individuals with of LDL-C >190 mg/dL• Individuals with diabetes mellitus aged 40–75 y with LDL-C 70–189 mg/dL• Individuals with LDL-C 70–189 mg/dL and estimated 10-y ASCVD risk ≥7.5%
Cholesterol treatment targets	Lipid-lowering therapy is initiated and titrated to achieve target LDL-C levelsLDL-C targets:• CHD and risk equivalents (10-y risk>20%): <100 mg/dL• Multiple (≥2) risk factors (10-y risk, ≤20%): <130 mg/dL• 0–1 risk factor: <160 mg/dL	The decision to initiate statin therapy and intensity of statin dosing should be based on the ASCVD risk and not on lipid targets. Clinicians can recheck LDL-C levels at 4–12 weeks after initiation to assess response to therapy and adherence and intensify treatment to achieve at least a 50% reduction
Dosing of statins	In most cases, the statin should be started at a moderate doseThe starting dose of statin will depend on the baselineLDL-C level	Initiate and maintain maximum tolerated statin intensity for patients <75 years oldModerate statin intensity for patients older than 75 years

Note: ATPIII: Third Report of the National Cholesterol Education Program Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III).

Abbreviations: ACC/AHA, American College of Cardiology/American Heart Association; LDL-C, low-density lipoprotein cholesterol; y, year.

In 2011, the California Safety Net Institute (SNI) and the California Association of Public Hospitals and Health Systems (CAPH) launched a pay-for-performance incentive program in California’s 21 public hospital systems to strengthen care delivery throughout their entire health systems and make high-quality care more accessible.11 The program, entitled the Delivery System Reform Incentive Program (DSRIP), involved health systems that care for over 2.5 million of California’s most vulnerable populations. Through this program, all public hospital health systems in California began reporting quality metrics to the state health department and the Centers for Medicare and Medicaid Services. After the implementation of DSRIP, the 21-member health systems jointly and voluntarily decided to set two QI goals, which they called the “Big Aims”, in 2012. One target was to improve cholesterol control in diabetes patients, defined as LDL-C level <1 g/L (100 mg/dL), such that >62% of eligible patients would meet this target consistent with the 90th percentile for Medicare Preferred Provider Organizations.

Together with the University of California San Francisco’s (UCSF) Center for Vulnerable Populations, CAPH and its member health systems worked to reach this goal as a part of the collaborative Public Healthcare Evidence Network and Innovation eXchange,12 an implementation science network which facilitates sharing of information among hospitals participating in the DSRIP. Using professional knowledge about key leaders and personnel, the SNI successfully recruited 40 QI leaders from the 21 public hospital systems to lead and coordinate implementation of the cholesterol management initiative in their respective health systems. Hospital systems spent the next year conducting QI efforts, which varied by site. Initiatives included upgrading data infrastructure to better capture laboratory data, implementing clinician education specifically related to LDL measurement in diabetes patients, and having medical assistants conducting patient inreach and outreach to schedule laboratory testing and follow-up visits.

To our knowledge, there has been no previous study to systematically explore how QI leaders on the frontline at public hospitals and safety-net health systems have reacted to the new guideline. In this study, we conducted six focus groups with representatives of California’s 21 public hospital systems to examine their reactions to the new 2013 ACC/AHA cholesterol treatment guideline for the purpose of exploring how the new guidelines affected their current cholesterol QI activities, and elicit their intentions or insights for future QI and performance measurement initiatives to integrate the new guideline into practice.

Materials and methods

Approach

Participants were recruited from the Public Health care Evidence Network and Innovation eXchange Network, comprising the member systems of CAPH and its QI affiliate, SNI – representing four University of California and 17 county-owned or county-operated health care systems. Members of CAPH/SNI provide services in 15 counties where >81% of Californians reside, delivering care to 2.5 million Californians. California’s public hospital systems provide >10 million outpatient visits each year in hundreds of primary and specialty care clinics.

Focus groups

UCSF investigators conducted six focus group interviews with all 40 CAPH QI leaders involved in ongoing cholesterol management initiative during a prespecified meeting. These 40 participants included QI personnel including frontline staff, managers, administrators, and clinicians who work on data management or performance reporting. Each participant gave written informed consent for the study (including audio recording and subsequent qualitative analysis and publication), and the UCSF Committee on Human Research reviewed and approved the protocol.

We designed the focus group guide to foster discussion on the effects of the change in cholesterol guidelines and its effect on current QI and performance monitoring initiatives. The interview guide included questions such as the following: 1) “In November of 2013 the AHA/ACC changed cholesterol guidelines; can you please tell us about these guidelines?” 2) “How will or does the new LDL guideline affect your performance measurement?” 3) “How do you prioritize between (your current cholesterol QI initiative) and the new guideline?” 4) “How feasible is it to make a performance measure based on the new guideline?” 5) “If you could make up your own cholesterol performance measure, what would it be?” The full focus group discussion guide is available in the online Supplementary material.

On average, each focus group comprised six participants representing three CAPH health systems (Table S1). Each focus group session was conducted by one of six UCSF investigators for durations of 60 minutes each. The full list of hospital systems is available in the online Supplementary material. All focus groups were audio-recorded and transcribed for analysis using the Dedoose qualitative analysis software.13

Qualitative analysis

We employed an integrated (inductive and deductive coding), team-based approach to thematic content analysis of the transcripts.14 A priori thematic categories were anticipated based on the interview questions (Table 2). Three authors conducted the initial qualitative content analysis as follows: 1) the primary author developed a list of themes after reading two transcripts; 2) two additional authors independently reviewed the two transcripts and the code list to achieve a consensus on the initial thematic coding; 3) the primary author used this code list as the guide for coding the remaining interviews and 4) made subsequent refinement as additional constructs emerged; and 5) the final list was reviewed by the remaining coauthors and discrepancies in coding were resolved by consensus.

Table 2

Summary and interpretation of thematic categories and subthemes

Thematic categories	Perception/attitudea	Feasibility of implementationb	System-level intentions for implementationc
Subthemes	• New guideline unclear and lacks direction for implementation• General sense of confusion on how to adopt the new guideline in practice	• Inadequate data infrastructure to move from LDL-C targets to a new cholesterol treatment performance measure• Limited funding and staffing to simultaneously adopt a new performance measure and undertake QI efforts based on the new guidelines• Lack of organizational infrastructure for disseminating and implementing guideline changes	• Abandonment of current cholesterol QI activities based on outdated LDL-C goals• Uncertainty and no commitment for new QI efforts to implement the new guidelines
Interpretation	Perceptions and attitudes toward the new guidelines impact on quality improvement were unfavorable	Existing structural inadequacies made system-level guideline implementation impractical for most safety-net health systems	Health systems’ intentions on how to implement the new guideline remained unclear

Notes:

Perception/attitude refers to participants’ perceptions, reactions, or attitude in regards to guideline’s impact on quality improvement.

Feasibility and capacity refers to participants’ perceptions on their ability implement the new guideline.

System-level intention refers to their plans or intent to undertake system-level coordinated efforts to implement the new guideline.

Abbreviations: LDL-C, low-density lipoprotein cholesterol; QI, quality improvement.

Results

Qualitative analysis of the focus group discussion uncovered four main themes: 1) unfavorable perceptions and attitudes relating to system-level implementation of the new guideline; 2) structural inadequacies that limit the feasibility of system-level implementation of the guideline; 3) uncertainty and limited commitment for undertaking coordinated efforts to integrate the new guideline into practice; and 4) insights or potential solutions to translating evolving guidelines into QI and clinical practice.

Perceptions and attitudes toward system-level implementation of the new guideline

The overall perception and attitude on implementation of the new guideline was unfavorable. The most common reaction was that the guideline did not provide adequate direction for implementation. There was a prevailing sense of confusion stemming primarily from a perceived lack of guidance for implementation and mechanisms for evaluating performance or quality. The participants reported a desire and need for more direction on how to implement the new recommendations at the system level. Below are representative quotes:

The guidelines aren’t very clear.

I feel like we need guidance [for guideline implemen tation].

So I think it just made it, without commenting on the guidelines, it just made it a challenge […] in terms of focusing every body’s attention on the same goal […]. I think that’s really where we are – is probably not having clarity of focus.

Themes related to feasibility of system-level guideline implementation

The participants highlighted several structural challenges that made translating the new guideline into clinical practice and performance monitoring very complex and impractical in their clinical settings. These challenges can be summarized into three themes which are as follows: 1) inadequate data infrastructure to move from LDL-C targets to a new cholesterol treatment performance measure; 2) limited funding and staffing to simultaneously adopt a new performance measure and undertake QI efforts based on the new guidelines; and 3) lack of organizational infrastructure for disseminating and implementing guideline changes.

Inadequate data infrastructure was a primary concern because most systems did not have a level of access to pharmacy prescribing and claims data that enabled capture of medication use or prescription data necessary to verify the increased use of statins that is recommended by the guideline. Another structural challenge repeatedly discussed in all the focus groups was limited funding. Health care administrators and QI leaders faced the task of improving performance on numerous quality indicators in addition to other priorities such as practice transformation, financial solvency, and medical education, with limited funding and resources dedicated to QI. These feelings were emphasized because of particularly tight funding streams available within safety-net health care systems, which often translated into insufficient staffing in several departments: Last, participants indicated a lack of standard mechanisms at their organizations for adopting new evidence into clinical practice. Mechanisms for disseminating new guidelines and educating providers on new evidence were “inconsistent” and inadequate. This concern was pervasive among participants in all six focus groups. Since scientific discovery and guideline development is iterative and cyclical, participants expressed a strong desire to have standard structures and mechanisms to disseminate and adopt evolving guidelines into practice.

[QI] is essentially an unfunded mandate […]. We have too many [quality] metrics right now […] we’re so busy just getting the data, validating the data, there’s not enough time for quality improvement. The same people have to do both.

We don’t formally have a clinical guideline steering committee or review committee. So when something comes up, people start calling each other and say, hey, do you want to get together and talk about this?

Themes related to system-level intentions for integrating the new guideline into practice

QI leaders from these health systems portrayed a high level of uncertainty and limited commitment for undertaking coordinated efforts to implement the new guideline. Participants expressed frustration with the lack of direction and complexity of operationalizing the new guidelines that may have contributed to a loss of momentum or impetus for cholesterol QI efforts at their institutions. These safety-net institutions were faced with the decision whether to continue their current QI efforts for improving cholesterol management based on LDL-C targets or develop new performance measures based on the new guidelines. There was a near consensus that they could not continue the current cholesterol QI efforts, because physicians did not accept metrics based on outdated guidelines. However, most of the systems did not have any plan or intention for coordinated efforts to implement the new cholesterol guideline into practice. When confronted with the question how they plan to adopt the new guideline, only one health system reported having a strategy of how they intend to implement the new guideline, as represented in the quote below: Otherwise, most participants conveyed a sense of bewilderment: Some participants expressed a reluctance to devote continued resources toward improving cholesterol performance because they did not know what cholesterol performance metric would eventually be adopted by the Centers for Medicare and Medicaid Services or private health insurers.

We’ll continue to do [the LDL] reporting, but we’ll do it in parallel to reporting on the statin use […]. We will totally focus our quality improvement on statin use rather than the LDL measure.

When the guidelines came out last year, the medical director called me and said ‘what am I supposed to do with this?’

We have not decided what, if anything, we’re going to do about the guideline change at the local level.

The other thing is that when you look at the measures [we are required to report under pay-for-performance policies] they are not changing right now.

Insights on strategies to translate changing guidelines into practice

Participants offered varying insights and suggestions with regards to appropriate performance measures and strategies to implementing the new guideline into practice. Discussions centered around two general approaches to translating guidelines into QI: 1) development of appropriate and feasible performance measures based on new guidelines and 2) establishment of organizational infrastructure for ongoing QI that can accommodate future changes in standards of care.

The first suggestion was that simple performance measures are inadequate. While one participant suggested “prescription pickups” of statins as a potential metric, a plurality of participants suggested a need for moving away from single-variable methods of measuring performance, with preference for more comprehensive measures. Simple performance measures neglect the reality that patients often have chronic conditions that commonly coexist. Therefore, many participants expressed the need for “holistic” measures that reflect the medical complexity of their patient population, in this case, patients with diabetes. In addition to more complex metrics, there was an even greater consensus for more flexibility choosing appropriate performance metrics and finding local solutions to QI. The second suggestion was that an emphasis should be placed on building capacity and support for QI. The focus groups ignited lengthy discussions on the cyclical aspect of incorporating new guidelines into practice in a way that preserves the focus on improving quality of care. These discussions highlighted a potential tension between QI and performance reporting. This led most participants to favor approaches that shift the focus away from additional performance measures. In lieu, participants expressed a need for more emphasis on developing organizational mechanisms for implementing change in clinical care.

Statins do not reduce blood glucose. Statins do not reduce your A1C. So I would like to see something a bit more holistic.

One thing I would like to see in the future is maybe a little bit more flexibility in how the metrics are defined so that the metric can accommodate changes in guidelines over time.

If you look at the number of metrics, it’s too many […] so there is that issue. In my view, there could be less.

We have too many [quality] metrics right now […] we’re so busy just getting the data, validating the data, there’s not enough time to improve, for quality improvement, the same people have to do both.

Sometimes it isn’t the actual measure itself. Sometimes, it’s very much about laying down the infrastructure for doing change. I think we really have to embrace that this is a long-term project of changing behavior, changing culture, and changing process and workflow.

Discussion

Our qualitative analysis of focus group discussions with QI leaders in California safety-net health systems found that the 2013 ACC/AHA cholesterol treatment guidelines created a general sense of confusion and lack of direction for centralized QI efforts in the management of cholesterol. Overall, participants conveyed unfavorable perceptions and attitudes regarding the guidelines’ potential impact on cholesterol QI efforts. They cited funding, organizational, and data infrastructural inadequacies that made system-level implementation of the new guidelines impractical in most safety-net health systems. Subsequently, most health systems remained undecided on how to implement the new guidelines at the system level.

Other investigators have described barriers to implementing evidence-based guidelines,15 including poor physician knowledge, time pressure, and inadequate data infrastructure. Our study findings corroborate the importance of adequate educational and data infrastructure in guideline implementation. Moreover, our work advances the literature by using qualitative analysis of focus groups to understand real-time reactions to the recent cholesterol guideline in safety-net health care institutions which face unique challenges including inadequacies in funding and technological infrastructure to facilitate system-level implementation of new evidence. Furthermore, our study also explored potential solutions to the challenge of adopting new evidence in clinical practice – frontline QI professionals at California safety-net health systems expressed a need for guidelines with clearer directions for implementation, more flexible and holistic performance measurement systems, and organizational and data infrastructures that facilitate consistent adoption of new guidelines into clinical practice.

Implications for guideline implementation and performance measurement

The dynamic of guideline changes and subsequent delay in updating performance reporting requirements can lead to frustration and further predispose loss of momentum for QI. The challenge of implementing guideline changes is especially burdensome in safety-net health care institutions which typically have suboptimal personnel and technology infrastructures needed for quality measurement and improvement.3,4 Moreover, absence of electronic health records with the capacity to develop disease-specific registries that facilitate ongoing quality performance monitoring and patient outreach often threatens sustainability of QI efforts undertaken in these clinical settings.16 In response, our focus group participants called for a greater emphasis on developing organizational and infrastructural capacity to accommodate change and undertake QI activities toward high-quality care. This strategy would potentially alleviate challenges and improve health systems’ ability to implement new guidelines in practice.

The reactions and insights highlighted in our study are largely consistent with various approaches that researchers and thought leaders have proposed in the literature.15,17–19 For example, Werner and McNutt20 and Goitein21 have proposed policy approaches that would incentivize institutions to invest in and develop their own QI programs tailored to local needs. Goitein suggested that such programs might be led by clinicians with dedicated, paid time, have their own budget, and be given centralized data, statistical, and technology support.

Implications for guideline development

Our findings also raise important considerations for guideline development. Focus group participants expressed significant frustration with the “lack of direction” and complexity of operationalizing the new cholesterol guidelines into performance measurement and QI. This sentiment echoed the concerns that have been raised in recent expert opinions and commentaries in the literature. In a commentary criticizing the guideline’s development process, Nissen9 stated that the guidelines left “practicing physicians in a bewildered state, eroding the confidence required for successful implementation”. We did not design this analysis to examine the evidentiary merits of the 2013 cholesterol guideline contents but focused on frontline reactions to the guidelines and its impact on implementation and QI efforts. Our findings highlight a key limitation pertinent to not just the recent cholesterol guideline, but also the growing trend of practice guidelines that answer narrowly defined questions, and attempt to avoid making recommendations in areas where evidence from clinical trials is lacking.22,23 This guideline development process leads to guidelines with limited scopes that provide little guidance for implementation, especially in diverse and understudied populations and settings. Inevitably, and perhaps appropriately, guidelines may leave a gap that medical societies, health care institutions, or providers must fill.8 It remains unclear what body should assume the role of providing guidance for guideline implementation.

Investigators and thought leaders have proposed varying frameworks that include prioritizing fewer performance measures,24 de-emphasizing performance reporting,20,21 or modifying guideline development to include “ implementability” content.1,17 Concerns regarding the new cholesterol guideline’s “clarity” and “lack of direction” for successful implementation should prompt a broader conversation about how best to bridge the gap between national guidelines, implementing evidence-based medicine in clinical practice and systems approaches to improve quality of care.

Strengths and limitations

To our knowledge, this is the first study to examine front-line perceptions of the change in cholesterol guidelines at public health care systems. Despite its strengths, this study does have limitations. Our analysis could not adequately examine differences based on the type of health care systems, such as large versus small or academic versus non-academic, because interview transcripts did not map participant responses to their individual identities. Larger systems with greater infrastructural capacity (eg, advanced electronic health record systems, data personnel, research, or QI faculty) may be more adept at implementing the new guidelines. However, our results are particularly salient because the same populations facing cardiovascular disease disparities (ie, low-income, racial/ethnic minority populations) disproportionately receive care in safety-net health care settings similar to those represented in our focus groups. Although our results suggested a possible loss of momentum in QI for cholesterol management, our qualitative methods were not designed to determine whether there was an actual change in quality of care. Although focus group discussions showed that participants had a good general understanding of the guidelines and their main recommendations, incomplete understanding or misconceptions about the guidelines that we could not detect may have influenced participants’ reactions to the guidelines.

Conclusion

The experience of California’s health care safety-net in grappling with performance measurement and QI in the wake of new cholesterol treatment guidelines highlights important challenges in translating new evidence into clinical practice and suggests a need for policymakers to consider a new approach for improving the “implementability” of new guidelines and promoting or rewarding higher quality of care. Value-based payment or other policies that shift focus from reporting quality measures toward emphasizing participation in processes to improve care merit continued exploration.

Data availability

The datasets generated during and/or analyzed during the current study are not publicly available but are available from the corresponding author on reasonable request.

Supplementary materials

Low-density lipoprotein (LDL) focus group guide

LDL breakout session – 45 minutes total

Priority focus is on effects of guideline change and its effect on Big Aims

Prompt: We want to talk with you about the LDL Big Aim. Specifically your thoughts around the Aim itself with regards to performance measurement and any challenges you may have had. We also want to see what your thoughts are around the changes to the LDL guidelines from the American Heart Association and the American College of Cardiology.

What is the Big Aim for cholesterol? (5 minutes)

When the Big Aims were first announced what were your thoughts about reaching the LDL Big Aim? (10 minutes)

Did it seem attainable?

What were the barriers to achieving the Big Aim?

Performance measurement demands – how do they relate to guidelines? Better health?

In November of 2013, the AHA/ACC changed cholesterol guidelines; can you please tell us about these guidelines? – Facilitator to clarify if necessary using definition below. (5 minutes)

How will or do the new LDL guidelines affect your performance measurement? (10 minutes)

The guidelines are vague – How does this affect performance measurement?

How do you prioritize between the Big Aim and the new guideline? (5 minutes)

How feasible is it to make a performance measure based on the new guidelines? (5 minutes)

Why or why not?

If you could make up your own cholesterol measure what would it be? (5 minutes)

LDL Big Aim definition

GOAL: By 6/30/15, achieve the 2011 HEDIS Medicare PPO 90th percentile, 62.2% LDL Control (LDL<100)

Table S1

Characterization of interviewees

Health system	Interviewee(s) title	Number of interviewees	Clinician type represented (if applicable)
1	Chief Medical Officer/Medical Director	1	MD
2	Chief Medical Officer and Director of Quality Services	2	DO; and non-clinician
3	MD, Clinical Associate Professor of Medicine, Mammography Champion	1	MD
4	Medical Director, Quality Improvement	1	MD
5	Medical Director of Quality (1) and Chief Medical Informatics Officer (1)	2	MD
6	Chief Medical Officer (1) and Chair Department of Primary Care (1)	2	MD
7	Medical Home Manager (1) and Medical Director, Ambulatory Services (1)	2	RD, MPH; and non-clinician
8	Deputy Director, Ambulatory Administrator (1) and Ambulatory Care Medical Director (1)	2	MD; and non-clinician
9	Chief Information Officer (1) and DSRIP Project Director (1)	2	Non-clinician; and non-clinician
10	Senior Deputy Director, Ambulatory Care Services	1	Non-clinician
11	Chief, Division of Primary Care (1) and Director of Primary Care and Community Health Services (1)	2	MD; non-clinician
12	Executive Medical Director, Primary Care	1	MD
13	Medical Director, PCMH	1	MD
14	MD, Mammography Champion – Associate Medical Director	1	MD
15, 16a	Chief Medical Officer	1	MD
17–20a	MD, Mammography Champion – Director of Women’s Health Programs and Innovation	1	MD
Total systems represented		Total interviewees	Total interviews
20		23	16

Note:

These distinct systems jointly report performance measures to the state health department as part of their consolidated DSRIP plan.

Abbreviation: DSRIP, Delivery System Reform Incentive Program.