Prioritizing Evidence-based Interventions for Dissemination and Implementation Investments: AHRQ's Model and Experience.

BACKGROUND: The Agency for Healthcare Research and Quality (AHRQ) is mandated to implement patient-centered outcomes research (PCOR) to promote safer, higher quality care. With this goal, we developed a process to identify which evidence-based PCOR interventions merit investment in implementation. We present our process and experience to date.
MATERIALS AND METHODS: AHRQ developed and applied a systematic, transparent, and stakeholder-driven process to identify, evaluate, and prioritize PCOR interventions for broad dissemination and implementation. AHRQ encouraged public nominations, and assessed them against criteria for quality of evidence, potential impact, and feasibility of successful implementation. Nominations with sufficient evidence, impact, and feasibility were considered for funding.
RESULTS: Between June 2016 and June 2018, AHRQ received 35 nominations from researchers, nonprofit corporations, and federal agencies. Topics covered diverse settings, populations, and clinical areas. Twenty-eight unique PCOR interventions met minimum criteria; 16 of those had moderate to high evidence/impact and were assessed for feasibility. Fourteen topics either duplicated other efforts or lacked evidence on implementation feasibility. Two topics were prioritized for funding (cardiac rehabilitation after myocardial infarction and screening/treatment for unhealthy alcohol use).
CONCLUSIONS: AHRQ developed replicable criteria, and a transparent and stakeholder-driven framework that attracted a diverse array of nominations. We identified 2 evidence-based practice interventions to improve care with sufficient evidence, impact, and feasibility to justify an AHRQ investment to scale up practice. Other funders, health systems or institutions could use or modify this process to guide prioritization for implementation.

BACKGROUND

The Agency for Healthcare Research and Quality (AHRQ) envisions an efficient model of care and research, with patients at the center of both. AHRQ’s Care and Learn model describes a learning health system where evidence, produced in conjunction with patient care, is then used to improve the ability of the health system to care for patients.1 AHRQ is specifically charged to disseminate findings from patient-centered outcomes research (PCOR) conducted by the Patient-Centered Outcomes Research Institute (PCORI), the National Institutes of Health (NIH), and others. AHRQ’s challenge has been to identify and choose among those evidence-based interventions that improve patient-centered outcomes, but are not yet in common practice.2 Closing the knowledge-to-practice gap requires dedicated effort and funding.

To respond to this charge, AHRQ developed a PCOR Dissemination and Implementation (DI) Initiative to invest a portion of the AHRQ-directed funds from the Patient-Centered Outcomes Research Trust Fund. In 2015, an AHRQ workgroup established goals and a framework for the PCOR DI Initiative based on principles of transparency, consistency, and scientific rigor. AHRQ invited nominations of evidence-based interventions by any interested persons and, using structured criteria for strength of evidence, potential impact, and feasibility of implementation, winnowed nominations to identify targeted areas for investment in implementation activities.

The aim of this paper is to describe AHRQ’s process and experience in operationalizing the first 6 prioritization steps of the PCOR DI Initiative framework during its first 2 years. By describing our efforts to apply the criteria, and explaining our decision-making, others may learn from our experience. Our work can inform other funders, health systems, or institutions working to prioritize evidence-based interventions for DI into clinical practice.

MATERIALS AND METHODS

The PCOR DI Initiative workgroup included clinicians and researchers with experience in health services research, evidence-based practice, systematic reviews, and DI science from across the Agency. First, the group refined AHRQ’s 7-step PCOR DI Initiative guiding framework (Fig. 1). The framework was designed to identify interventions that were supported by a strong evidence base; would have a great impact on health outcomes that matter most to individuals, communities, and the broader public; and were feasible to implement in a variety of clinical settings. Within the workgroup, 2 teams focused on evaluation of evidence and impact and on evaluation of feasibility of DI.

FIGURE 1

Agency for Healthcare Research and Quality patient-centered outcomes research (PCOR) Dissemination and Implementation initiative 7-step prioritization framework.

The first step of the PCOR DI process was to get nominations of PCOR interventions from the public, including researchers, consumers, organizations, and other federal agencies (step 1 in Fig. 1). To do so, we hosted a public nomination submission portal on the AHRQ Web site. We actively promoted the nomination portal through AHRQ’s electronic newsletters, the National Cancer Institute’s Implementation Science Newsletter, targeted email outreach to stakeholder organizations, and in person at research conference presentations. In addition, AHRQ worked with PCORI to identify topics from their portfolio of funded PCOR projects. These efforts generated broad awareness among a variety of stakeholders who understood the need to support implementation of PCOR. We also considered AHRQ’s portfolio of systematic reviews from the Effective Health Care Program and the United States Preventive Services Task Force for potential nominations. Nominators were asked to provide the supporting PCOR evidence and information about impact, feasibility, and potential implementation strategies.

For step 2, 1 team member tracked nominations and evaluated complete submissions against the minimum criteria. This ensured that the intervention improved patient-centered outcomes and that results had been published in peer-reviewed literature (step 2 in Fig. 1). Nominations that met minimum criteria went forward for evidence and impact assessment.

During step 3, we assessed the strength and quality of evidence, and the potential impact of the nominated intervention on patient-centered outcomes (step 3 in Fig. 1). To establish the criteria for these assessments, AHRQ reviewed the literature and gathered input from a diverse stakeholder group with expertise in assessing evidence and prioritizing research. Our final evidence criteria were based on the quality, consistency, precision, and effect size of the published research, consistent with criteria used by the AHRQ Evidence-based Practice Centers program and other similar organizations.3,4 We rated strength of evidence as low, moderate, or high. Similarly, criteria for impact built upon the literature and included measures of population burden (encompassing both the number of persons affected and the severity of outcome), practice gap, and potential to relieve disparities5 (Table 1). Only those nominations rated as moderate or high for both strength of evidence and impact went forward for further consideration.

TABLE 1

Summary of Criteria for Evidence, Impact and Feasibility Criteria, and Justification for Structured Implicit Review

The goal of step 4 was to identify which nominations could be feasibly implemented into widespread clinical practice, and/or at the national level. AHRQ searched the literature for existing criteria to use or adapt for assessing feasibility of interventions for scaling up and spreading in order to prioritize interventions for investment. The term “feasibility” often refers to whether an intervention is acceptable to users, or fits within a local workflow. Since we did not identify a comprehensive set of criteria to meet our purposes, we developed our own set of implementation feasibility criteria based on a literature review, key informant interviews, and a technical expert panel (Table 1). Team members used a standardized feasibility tool to frame their feasibility report. On the basis of these criteria, we rated feasibility as low, moderate, or high.

For both the evidence/impact assessment (step 3 in Fig. 1) and the feasibility assessment (step 4 in Fig. 1), we did not have explicit benchmarks such as “evidence: must improve outcome by at least X% over usual care” or “impact: must affect at least N persons in the USA.” Instead, the teams used structured implicit review to assess a heterogeneous set of topics against prespecified criteria in a transparent, reliable way (Table 1).6 We used an iterative process during which the team discussed the assessment to achieve consensus on each criterion rating, as well as on the overall final assessment.

The full PCOR DI Initiative group then considered implementation approaches for nominations rated as highly feasible (step 5 in Fig. 1). During this step, AHRQ engaged stakeholders with expertise and experience relevant to the specific nomination topic. The group consulted with in-house experts and external stakeholders to ensure that we consider which approaches might succeed, given the health care landscape and most significant barriers and facilitators to implementing complex interventions. Further, we wished to avoid duplication of other efforts.

Nominations that passed step 5 were developed into project concepts and presented to senior leadership for funding decisions (step 6 in Fig. 1). Senior leaders were involved to assure that the selected projects align not only with AHRQ’s mission, mandate, and priorities, but also with its expertise and resources. The final step, step 7, incorporated evaluation into the development and funding of each DI project to ensure that AHRQ’s work improved the delivery and quality of care and contributed new knowledge on implementation.

To aid understanding of how we operationalized and applied our framework, we now describe and illustrate the results of our decision process.

RESULTS

Nominations

In the first 2 years of the DI initiative (June 2016 to June 2018), AHRQ received 35 complete nominations through the Web site nomination portal (step 1).7 Nominators included academic researchers, nonprofit corporations (such as a patient safety organization and a professional medical society), individuals, PCORI, and federal agencies. Federal agency nominators included the Uniformed Services University of the Health Sciences, the Center for Disease Control and Prevention, the Food and Drug Administration, the Centers for Medicare & Medicaid Services, the NIH/National Cancer Institute, and AHRQ. Nominations covered diverse settings (primary care, emergency, inpatient, rehabilitation, and home), populations (pediatric, adolescent, adult and elderly), and clinical areas.

Of the 35 complete nominations received during our prioritization process, 7 nominations did not meet our minimum criteria (step 2). Five of those did not describe PCOR. For example, 1 nomination proposed that AHRQ and Centers for Medicare & Medicaid Services change a patient safety indicator based on a secondary data analysis. Two nominations did not have peer-reviewed evidence (one reported a local quality improvement project, and another planned to implement a new pain scale).

Evidence and Impact Assessment

There were 28 unique PCOR nominations which met minimum criteria and were evaluated for strength of evidence and impact (step 3); of these, 10 did not have enough published evidence that the clinical intervention was effective (Fig. 2). For example, 1 nomination had no published literature on the proposed multicomponent intervention as a whole. When we assessed the evidence for the individual components, each had low strength of evidence based on the quality of the studies, and applicability of the intervention to the population and setting of interest. We concluded that there was insufficient evidence for the multicomponent intervention. For another, we found that the literature base was composed of multiple poor to fair quality observational studies, and the effect size could not be determined.

FIGURE 2

Diagram describing how nominations were winnowed through the assessment phases of the AHRQ patient-centered outcomes research Dissemination and Implementation (D&I) initiative prioritization process. Numbers as of June 2018. AHRQ indicates the Agency for Healthcare Research and Quality; PCOR, patient-centered outcomes research.

Two nominations were judged to have insufficient potential for impact. For example, 1 nominated intervention resulted in the patient’s body mass index (BMI) moving about 0.06 SDs closer to the mean (z score). However, the literature reports that clinically significant improvement occurs after BMI z score reductions of 0.15–0.20 U.8 Thus the improvement in BMI z scores reported in this nomination might not have clinical impact. In another case, the population burden and gap in care were unclear: the number of patients with the condition and proportion who were eligible for the nominated intervention were unknown.

Feasibility Assessment

Sixteen interventions with moderate to high evidence/impact were assessed for feasibility (step 4), 12 of which did not meet feasibility criteria. In 4 of these 12 cases, implementation would have duplicated efforts of other funders; 8 did not go forward due to incomplete or unconvincing evidence of feasibility of an AHRQ investment. In 1 case, a new guideline recommended a change in practice, but recent performance data were unavailable precluding our ability to assess the potential benefit of a widespread implementation effort at this time. In 2 cases, available data indicated that practice was already changing in the correct direction to improve care (the gap between ideal and actual care was decreasing), suggesting that additional AHRQ investment may not be needed. In another case, implementation strategies for the proposed intervention were actively being studied, and results would be available in the near future.

One intervention required complex decision-making for both clinicians and patients, within a rapidly changing landscape of clinical treatments. To clarify potential implementation approaches for this topic, we convened an in-person stakeholder meeting in partnership with PCORI. The stakeholders underscored the complexity of the topic and identified the need for additional research on effective multicomponent and multilevel implementation strategies. Their discussion informed AHRQ’s decision that this topic was not yet ready for widespread dissemination or implementation.

All 8 of those nominations that we judged “not feasible for AHRQ at present” had moderate to high scores for evidence and impact, and thus may be considered in the future as more evidence becomes available.

Funding Decisions

Of the 4 nominations that met feasibility criteria, 1 is still under review. Three fit AHRQ’s mission and mandate were deemed appropriate for funding in 2019 (Fig. 2). Two of these 3 were closely related topics and were combined into a single funding announcement: screening and brief intervention for adults with risky drinking,9 and medication-assisted treatment for those with alcohol use disorder.10 The Request for Funding Announcement (RFA-HS-18-002) required investigators to propose a multicomponent intervention to increase the uptake of screening and management of unhealthy alcohol use in primary care. The third nomination resulted in a funded contract11 to support scale-up and spread of an evidence-based implementation strategy to increase referral, enrollment, and retention in cardiac rehabilitation after eligible cardiac events.12–14

DISCUSSION

Like most organizations, AHRQ faces the dilemma of having multiple worthy areas for potential investment but limited resources. In our case, there are numerous PCOR evidence-based interventions that meet our mission to “make health care safer, higher quality, more accessible, equitable, and affordable, and to ensure that the evidence is understood and used.”15 To identify and prioritize which interventions might merit investment, AHRQ developed a transparent, stakeholder-driven framework, and criteria for assessing the strength of evidence, impact, and feasibility that attracted a diverse array of nominations in its first 2 years.7 We created a public nomination process for stakeholders to identify PCOR findings on our Web site7 and criteria for assessing the strength of evidence and impact.

The strengths of our approach include our public nomination process, flexibility, and alignment with other frameworks. Using a public web portal to solicit nominations increased the transparency of our processes, and our promotion strategy allowed us to target researchers, research funders, and other stakeholders. Another strength to our process is its flexibility: had we not found interventions that fit our criteria, we could defer funding, and engage with stakeholders to identify more nominations, or wait for new implementation evidence on the 8 topics that were not quite ready to disseminate broadly.

The framework that AHRQ devised aligns with other published approaches for prioritizing topics based on evidence and impact. For example, like Maciosek et al,5 we used estimates of population burden and practice gap to measure impact when setting priorities to scale up preventive health services. In this case, the authors worked from a prescribed list of preventive service interventions. However, we could not limit ourselves to this approach because AHRQ is not limited to a clinical area and there is a vast array of evidence-based interventions to choose among. For example, the NIH Office of Disease Prevention lists 571 evidence-based interventions related to healthy People 2020 objectives alone.16 Since its inception in 2012, PCORI has funded 620 projects, about one third of which are completed.17 Therefore, AHRQ used a public nomination strategy, supplemented by outreach to federal partners and internal experts, to identify topics with the greatest potential and stakeholder interest.

The AHRQ framework complements the Promoting Action on Research Implementation in Health Services (PARiHS) framework, which aids implementation approaches and evaluation when organizations have already selected the topic to be promoted into practice.18 Specifically, PARiHS authors recommend that strategies for implementation require careful planning and need to include criteria to evaluate the impact of the intervention. AHRQ developed a structured process to apply our feasibility criteria, and we describe our deliberations to consider which approaches fit the health care landscape and stakeholder needs. In prioritizing topics, we also considered fit with the Agency’s priorities, current resources, and expertise.

Unlike some other approaches, our framework did not define benchmarks for evidence, impact, or feasibility, but instead used a structured implicit judgment strategy. This required us to struggle as a team to assess nuances in effect sizes, practice gaps, and to clearly define “evidence that the intervention improves patient-centered outcomes” and “evidence that the intervention is ready to implement.” As can be seen in our discussion of feasibility, the impact of an intervention was often reviewed again under feasibility, with a focus on “what is AHRQ’s potential impact if we implement this strategy?” We used an iterative process, valuing team input, until we reached consensus from team members, and support from senior leaders. This required an investment of time and training for the team. With practice, we became more facile at identifying which topics were more likely to be suitable for implementation.

We successfully attracted a diverse array of nominations in the first 2 years of the initiative. Using the framework and criteria, we prioritized 3 nominations into 2 topics for DI investments (Table 2). Both impose a significant population burden. About 30% of US adults demonstrate unhealthy drinking habits, and 6% have alcohol use disorder.19 Close to 1 million Americans affected by cardiac events each year are eligible for cardiac rehabilitation.20 For both of these conditions, there are effective evidence-based interventions that are not being used optimally. Screening and counseling are effective,9 but fewer than 30% of adults are screened for alcohol use in primary care, and fewer than 20% of heavy drinkers are even counseled to reduce intake.21 Cardiac rehabilitation is an effective intervention that reduces cardiovascular mortality by nearly 30%, risk of hospital admissions by 31%, and improves health-related quality of life.12 Cardiac rehabilitation aligns with Million Hearts 2.0, but only 18%–30% of affected individuals receive this service.22,23

TABLE 2

Comparison of Intervention Topics Prioritized by AHRQ

For the cardiac rehabilitation topic, AHRQ identified a feasible strategy (automatic referral for cardiac rehabilitation with liaison services14) and will invest in national scale-up and spread of the Million Hearts Cardiac Rehab Change Package. For unhealthy alcohol use, we did not find a strong body of evidence for any single effective implementation strategy that improves diagnosis and management of unhealthy alcohol use in primary care. Therefore, grantees will be able to test new implementation strategies in this area, building additional evidence that can then be used to further increase uptake of these evidence-based practices.

Consistent with its “Care and Learn” model,1 AHRQ requires that its DI investments include an evaluation in order to contribute to the evidence on implementation. Each of these funded projects is expected to improve care for individuals, produce learning for primary care practices, hospitals, and health systems, and evidence for both AHRQ and the larger health care community on implementation. We expect to learn how these implementation strategies worked for different health care settings and populations. In addition, planned evaluation work may result in further adaptations to our framework and criteria.

We acknowledge some limitations to the process we developed. First, we relied on public nominations. However, we received many nominations on diverse topics from a range of stakeholders. Although we widely promoted the opportunity for nominations, stakeholder involvement was limited by the extent of our outreach. In addition, our nomination process required altruism, since it is not a pathway to funding for the nominator: when AHRQ chose to disseminate a topic, we publicized the funding announcement and awarded contracts and grants using our established rigorous processes.

In support of its mission, AHRQ established criteria and a rigorous prioritization framework which identified 2 stakeholder-nominated, evidence-based practices that are ready for further DI. We present concrete examples to demonstrate our journey from theory to practice. Others could use the tools and resources that we have developed to adapt this model to guide their investments in implementation.