Performance Measure Development, Use, and Measurement of Effectiveness Using the Guideline on Mechanical Ventilation in Acute Respiratory Distress Syndrome. An Official American Thoracic Society Workshop Report.

Guideline implementation tools are designed to improve uptake of guideline recommendations in clinical settings but do not uniformly accompany the clinical practice guideline documents. Performance measures are a type of guideline implementation tool with the potential to catalyze behavior change and greater adherence to clinical practice guidelines. However, many performance measures suffer from serious flaws in their design and application, prompting the American Thoracic Society (ATS) to define its own performance measure development standards in a previous workshop in 2012. This report summarizes the proceedings of a follow-up workshop convened to advance the ATS's work in performance measure development and guideline implementation. To illustrate the application of the ATS's performance measure development framework, we used the example of a low-tidal volume ventilation performance measure created de novo from the 2017 ATS/European Society of Intensive Care Medicine/Society of Critical Care Medicine mechanical ventilation in acute respiratory distress syndrome clinical practice guideline. We include a detailed explanation of the rationale for the specifications chosen, identification of areas in need of further validity testing, and a preliminary strategy for pilot testing of the performance measure. Pending additional resources and broader performance measure expertise, issuing "preliminary performance measures" and their specifications alongside an ATS clinical practice guideline offers a first step to further the ATS's guideline implementation agenda. We recommend selectively proceeding with full performance measure development for those measures with positive early user feedback and the greatest potential impact in accordance with ATS leadership guidance.

Contents

Overview

Methods

Results

Overview of GI Tools

PM Creation from CPGs

PM Framework in Practice

Rationale for guideline selection

Rationale for recommendation selection

Suggestings for scoring

Target audience and pilot testing

PM Implementation

Challenges and Future Plans

Conclusions

Overview

Guideline implementation (GI) tools are materials and information that support uptake of clinical practice guidelines (CPGs) by end users and are a key component of the American Thoracic Society (ATS) GI strategy (Figure 1). Performance measures (PMs) derived from CPGs are a type of GI tool that can positively influence clinician behaviors. However, many PMs have limited impact because of design and development flaws, such as being based on weak or insufficient scientific evidence or failure to pilot test in real-world settings before widespread application. The ATS convened a workshop in 2012 proposing a framework for PM development. This follow-up workshop in 2018 reviewed and applied the previous PM framework to the 2017 ATS/European Society of Intensive Care Medicine/Society of Critical Care Medicine (ATS/ESICM/SCCM) mechanical ventilation in acute respiratory distress syndrome (ARDS) CPG to create an actual PM and advance the ATS’s efforts in GI tool development. Key conclusions from this workshop include the following:ATS CPGs are rigorously developed, high-quality documents that define evidence-based clinical practices for the three pillars of the ATS: pulmonary, critical care, and sleep medicine. CPGs are a fundamental resource that informs healthcare delivery, evaluation, and quality improvement (1); yet, their use is inconsistent, leading to suboptimal patient care and outcomes (2–5). Thus, ensuring that its CPGs are adopted and implemented in real-world clinical settings is a top priority for the ATS. The ATS Executive Committee envisions a process (Figure 1) in which 1) guidelines are developed and disseminated, 2) GI tools are created, and 3) GI tools are used and their effectiveness is measured.

Figure 1.

The American Thoracic Society (ATS) envisions a three-step strategy to increase the adoption of clinical practice guideline recommendations. Each step is defined by an action, a derivative product, and a responsible ATS committee leading the effort. DDIC = Documents Development and Implementation Committee; PRS = Program Review Subcommittee; QIIC = Quality Improvement and Implementation Committee.

Appropriately developed PMs are a valuable tool to advance the ATS’s vision of CPG dissemination and adoption.

Professional societies infrequently develop PMs, despite these societies’ charge to advocate on behalf of the patients and clinicians whom PMs most directly affect.

The 2012 ATS PM development framework is consistent with the 2016 Guidelines International Network (G-I-N) reporting standards for guideline-based PM development.

The ATS PM development framework is outlined in Table 1. Key tenets include the requirement that candidate recommendations be based on only strong recommendations with at least moderate-quality scientific evidence via the Grading of Recommendations Assessment, Development and Evaluation (GRADE) schema. Feasibility of implementation and consideration of unintended consequences are required. Pilot testing of PMs is a requirement of any ATS PM destined for submission and endorsement by the National Quality Forum (NQF).

Table 1.

American Thoracic Society approach to guideline-based performance measure development using G-I-N reporting standards

G-I-N PM Reporting Standards (25) (2016)	ATS PM Development Framework (15) (2012, revised in 2018)
Composition of panel creating PM • Inclusion of multidisciplinary experts, stakeholders, and patient representatives	Panel includes: • Guideline developers and quality experts, ideally assembled and working synchronously with guideline creation • Broad stakeholder involvement
Guideline selection according to its: • Currency • Quality (rated by a validated tool such as AGREE II)	Select evidence-based guidelines that use GRADE methodology
Guideline recommendation selection • Strength and/or grade of recommendation that determines its eligibility	Candidate recommendations (by GRADE): • Strong recommendations • High- or moderate-quality evidence base
PM core attributes to consider and report: • Relevance • Scientific soundness • Feasibility	Required PM attributes: • Important • Scientifically sound • Feasible • Unintended negative consequences considered but outweighed by potential benefits
PM development process described in detail	PM development process described with detail and transparency
PM specifications: • Numerator • Denominator	PM specifications clearly defined: • Indicator statement • Numerator • Denominator • Exclusions • Scoring schema
PM intended use and target audience	PM intended use/audience: • Improvement of patient-centered outcomes • Self-improvement tool for clinicians • Broad application across diverse sites • Suitable for submission to NQF
PM piloting • Full description of piloting process • Rationale if no piloting done	PM piloting should be performed in multiple and diverse settings
Review and reevaluation of PM • Extent of PM use • Criteria for changing or stopping PM	Review and vetting by key stakeholders Evaluate impact, refine PM accordingly

Definition of abbreviations: AGREE II = Appraisal of Guidelines for Research and Evaluation II instrument; ATS = American Thoracic Society; G-I-N = Guidelines International Network; GRADE = Grading of Recommendations Assessment, Development and Evaluation; NQF = National Quality Forum; PM = performance measure.

Italicized text denotes additions to the 2012 framework by 2018 ATS workshop participants. Bold text denotes comparisons between G-I-N and ATS performance measures.

Adapted by permission from Reference 25.

Of three strong recommendations in the 2017 ATS/ESICM/SCCM mechanical ventilation in ARDS CPG (Figure 2), only the low–tidal volume ventilation (LTVV) recommendation was selected for PM development.

Figure 2.

Summary of the 2017 multisociety acute respiratory distress syndrome (ARDS) mechanical ventilation guideline, displayed by the GRADE-rated (Grading of Recommendations Assessment, Development and Evaluation) strength of each recommendation. Confidence levels reflect the quality of scientific evidence supporting the recommendation. “None” indicates that there was insufficient evidence to recommend either for or against the recommendation. The American Thoracic Society (ATS) recommends that only “strong” recommendations should be considered for performance measure development. ECMO = extracorporeal membrane oxygenation; ESICM = European Society of Intensive Care Medicine; HFOV = high-frequency oscillatory ventilation; PBW = predicted body weight; PEEP = positive end-expiratory pressure; Pplat = measured plateau pressure; SCCM = Society of Critical Care Medicine.

Specifications for the LTVV PM are shown in Figure 3. In considering the denominator for eligible patients, an ARDS definition that includes administrative billing codes and clinically mined data was proposed, given the known problem of underrecognition of ARDS.

Figure 3.

Using the revised 2012 American Thoracic Society performance measure development framework, workshop participants derived a performance measure based on the low–tidal volume ventilation recommendation of the 2017 mechanical ventilation in acute respiratory distress syndrome (ARDS) clinical practice guideline. A proposed indicator statement, numerator, denominator, and exclusions are included. ECMO = extracorporeal membrane oxygenation; PBW = predicted body weight.

PM pilot testing is a resource-intensive process that requires ongoing commitment by the lead organization and involved clinical sites, but it may yield mutually beneficial partnerships.

While the ATS develops mechanisms to support and sustain full PM development, it may apply its PM methodology to new CPGs through the step of defining specifications to create “preliminary PMs.”

Preliminary PMs will provide individuals and organizations with a ready-to-use GI tool. Full PM development, including pilot testing, should proceed for preliminary PMs with the most promising user feedback and the greatest potential impact in accordance with ATS leadership.

Step 1 of this vision occurs regularly. The ATS Program Review Subcommittee reviews and selects proposals that identify high-priority disease states or areas of clinical uncertainty, and the Documents Development and Implementation Committee (DDIC) oversees a panel that appraises and summarizes the evidence, formulates recommendations, and writes a guideline manuscript.

After publication of all CPGs, the ATS Director of Guideline Implementation oversees step 2 by working with the DDIC, the Quality Improvement and Implementation Committee (QIIC), the Patient and Family Education Committee, and others to develop GI tools, including editorials, clinical summaries, symposia, patient information fact sheets, patient videos, continuing medical education courses, pocket cards, electronic applications, Twitter chats, slide sets, and podcasts. These GI tools are housed on implementation webpages and aim to improve patient care by educating clinicians and patients about the evidence-based practices endorsed in CPGs.

Although necessary, GI tools focused solely on an educational strategy may be insufficient to change clinician behavior (6). PMs are metrics of adherence to a clinical process or metrics of actual clinical outcomes and represent a quality management implementation strategy (7) currently missing from the ATS’s portfolio of GI tools. Once a PM is defined, benchmarks can be set that establish standards of care to which individual clinicians and healthcare organizations are held accountable. Using PMs in this manner is a powerful impetus for influencing clinician behavior and healthcare system structure (8–10).

However, PMs must be developed correctly. Despite the rapid proliferation of PMs in the United States, a recent assessment of 86 Medicare Merit-based Incentive Payment System/Quality Payment Program PMs concluded that only 37% were rated as “valid” (11). A majority of the “not valid” measures lacked high-quality supportive clinical evidence. Similarly, 63% of polled physicians believed that current PMs do not accurately reflect the quality of care provided (12). Poorly designed PMs may inadvertently advocate clinical practices that have no effect, confer only small benefits relative to the resources expended, or cause harm (13, 14).

Amid these concerns, the ATS convened a workshop in 2012 to develop its own PM development framework with the goal of releasing ATS-endorsed PMs soon after the publication of newly published ATS CPG documents (15). Despite the publication of more than 28 CPGs since 2012, the ATS still does not routinely issue its own guideline-based PMs. Thus, to advance the ATS’s work in step 2 (GI tool creation, specifically PM development) and step 3 (GI tool use and measurement of effectiveness), in 2018, a follow-up PM workshop was convened. The intended audience included ATS committee members and leadership involved in advancing GI, clinicians and stakeholders who would be directly affected by this PM, and PM experts and professional society leaders also grappling with defining the role of professional societies in PM development.

Methods

In 2017, the DDIC, the QIIC, the Behavioral Science and Health Services Research Assembly, and the Critical Care Assembly jointly submitted a workshop proposal outlining their intent to use the creation of an ATS-authored PM as a demonstration project to both further the ATS’s strategic GI vision and refine the ATS’s approach to PM development. Workshop co-chairs (K.A.A., R.A.D., C.H.W., and R.C.H.) selected the 2017 mechanical ventilation in ARDS CPG (16) for PM development and invited speakers with international expertise in GI and PM development (A.R.G., S.H., and M.N.). The remainder of workshop participants provided clinical and content expertise in ARDS, implementation science, and quality improvement. The workshop proposal was approved and funded by the ATS Board of Directors. Potential conflicts of interest were disclosed, reviewed, and managed according to the policies and procedures of the ATS.

The full-day workshop was held on May 19, 2018, in San Diego, California. Speakers (A.R.G. and M.N.) reviewed types of GI tools and compared the PM creation framework proposed by ATS in 2012 (15) with that of the G-I-N. Next, participants engaged in four working sessions. First, they reviewed and revised the 2012 ATS PM creation criteria. Second, they applied the ATS framework to the 2017 ARDS mechanical ventilation CPG to select an appropriate recommendation for PM development. Third, they created a PM, including defining the indicator statement (15), numerator, denominator, and exclusion criteria. The final speaker (S.H.) reviewed strategies and challenges of implementing PMs in “real-world” settings. The fourth working session concluded the workshop by proposing strategies by which ATS could participate in PM advancement, both in general and specifically related to the new ARDS PM.

The workshop co-chairs and speakers authored the initial manuscript draft. Disagreements and incompletely defined components of the PM indicator statement were iteratively resolved via e-mail and teleconference discussion, with majority consensus obtained after the workshop date. The other workshop participants reviewed and edited the workshop report manuscript, which was followed by several cycles of external peer review and final approval by the ATS Board of Directors.

Results

Overview of GI Tools

The gap between CPG recommendations and actual clinical practice is well established (2–5). To understand how guideline developers might adapt the CPG product to better suit end users’ needs, Gagliardi and colleagues (17) identified features of CPGs that are associated with enhanced health professional adoption. Some characteristics were related to formatting, such as avoiding overly cumbersome narrative text or including a table of contents. Other characteristics pertained to enhanced CPG content, such as patient education materials or guidance for applying recommendations to specific patient groups. Collectively, any content within or derived from CPGs that support guideline uptake are termed “guideline implementation tools” (17). Categories of GI tools include materials that support patients or clinicians, implementation strategies, and evaluation efforts (18).

As confirmed by a 2016 Cochrane systematic review, GI tools developed and disseminated by developers represent an important way to influence clinician behavior and support guideline uptake (19). Unfortunately, when evaluated according to the applicability domain of the Appraisal of Guidelines for Research and Evaluation instrument, a metric of implementation potential and GI tools, CPGs published from 2008 to 2013 consistently score poorly and are without improvement compared with guidelines published in 2007 or earlier (20, 21). To assist guideline developers in generating GI tools, Gagliardi and colleagues used mixed methods approaches to generate a framework of types of GI tools for different target users and purposes (17), criteria for GI tool content and format based on international consensus (22), and practical considerations for developing GI tools based on the reported experiences of 26 GI tool developers in nine countries (23).

More recently, Liang and colleagues reviewed English-language CPGs on arthritis, asthma, colorectal cancer, depression, diabetes, heart failure, and stroke management published from 2010 to 2017 (18). Only 67.5% of CPGs included GI tools, the majority (51.5%) of which were designed for clinicians (guideline summaries, algorithms). Fewer (24.4%) were for patients (information sheets, self-management support), and fewer still (14.3% and 9.9%, respectively) were meant to support implementation (training materials, funding resources) or evaluation (audit tools, PMs). Of GI tools created by professional societies, only 1.5% were evaluation GI tools, suggesting a lack of engagement in this important implementation domain.

PM Creation from CPGs

Workshop participants first reviewed the ATS PM development framework proposed in 2012 (15). To minimize the problem of PMs disconnected from patient-centered outcomes or high-quality clinical evidence, the 2012 workshop participants proposed eligibility criteria based on a guideline recommendation’s GRADE (24) rating. Specifically, only strong recommendations based on high- or moderate-quality evidence should be chosen as PM candidates. The GRADE rating was chosen because it employs a high standard for deeming evidence to be of high or moderate quality, and the approach is used by the ATS to rate the strength of recommendations and the quality of evidence. In addition, recommendations should meet the Agency for Healthcare Research and Quality’s desirable PM attributes, including importance, scientific soundness, and feasibility (15). Once selected, the second step is to transform guideline recommendation(s) into indicator statements with defined numerator, denominator, and exclusion criteria. Finally, guidance is provided regarding how the PM will be scored, pilot tested, and evaluated, with the possibility of revisions based on these results (Table 1).

After discussing every step, the 2018 workshop participants endorsed the 2012 framework with several additions. First, consideration of unintended consequences was added to the list of attributes of candidate PMs. Second, “clarity of definitions” (of the indicator and its application) was stressed, such that target patient populations in CPGs should be explicitly and consistently defined. Third, ATS should rigorously develop PMs with the primary purpose of improving patient-centered outcomes and providing a self-improvement tool for clinicians. ATS PMs with sufficiently broad applicability and impact will be further developed and submitted for endorsement to the NQF. Finally, the 2018 workshop participants compared the ATS PM framework with the G-I-N reporting standards for guideline-based performance measurement development outlined in 2016 (25) (Table 1), and they found that it met essential requirements.

PM Framework in Practice

With this background, workshop participants applied the revised ATS PM development framework to the 2017 ATS/ESICM/SCCM mechanical ventilation in ARDS CPG (16).

Rationale for guideline selection

ARDS is a clinical syndrome of severe acute hypoxemia and noncardiogenic pulmonary edema resulting from inflammatory lung injury, with high mortality (26, 27). The 2017 mechanical ventilation in ARDS guideline was published within 1 year of the workshop and included six evidence-based recommendations, three of which were strong (Figure 2) (16). There are known gaps in the care of patients with ARDS (27), and the recommended mechanical ventilation strategies are under clinicians’ control. Collectively, these factors met the “importance” criteria for PM development and therefore served as a useful starting point for the workshop.

Rationale for recommendation selection

Higher positive end-expiratory pressure (PEEP), recruitment maneuvers, and extracorporeal membrane oxygenation were eliminated on the basis of not being strong recommendations. High-frequency oscillatory ventilation (HFOV) was rejected because the recommendation was against the use of HFOV, and recent evidence suggests that HFOV is rarely used even for severe ARDS (27).

Of the two remaining strong recommendations, LTVV was chosen over prone positioning on the basis of its greater potential to meet NQF suitability criteria as a PM (28). For example, LTVV applies to all patients with ARDS, whereas prone positioning applies only to patients with severe ARDS, defined as a ratio of arterial oxygen tension to fraction of inspired oxygen (PaO/FiO) less than 150 mm Hg (29). In addition, in considering the unintended consequence of either an LTVV or prone positioning PM being inappropriately applied to patients later discovered not to have ARDS, the potential harms of LTVV were judged to be less than those of prone positioning. Indeed, there is evidence that LTVV applied to patients without ARDS may have a beneficial or at least neutral effect (30, 31).

PM specifications

The committee concluded that the PM should follow the CPG recommendation as closely as possible, deviating only to improve the feasibility of the measure while not sacrificing its scientific soundness. Figure 3 illustrates the indicator statement and specifications of the PM.

Denominator

Significant effort was spent determining the proper denominator (Figure 3). In recognition that providing LTVV is not a single event and may be influenced by multiple clinicians, the denominator is defined as the number of opportunities to provide LTVV to patients with ARDS rather than as the number of patients eligible to receive it. Tidal volume and plateau pressure should be reported twice daily (separated by at least 6 h) within the first 24, 48, and 72 hours from onset of mechanical ventilation. Twice-daily reporting balances avoiding an overly burdensome reporting requirement with incentivizing clinicians to adjust ventilator settings until reaching LTVV goals. It also attempts to capture different team members’ performance on day and night shifts. The 72-hour time window is based on prior research showing that 93–97% of mechanically ventilated patients with ARDS meet ARDS diagnostic criteria within 48 hours of intubation and that 76% of patients with ARDS meet criteria at the onset of mechanical ventilation (27, 32). Thus, applying the PM to this early yet discrete time window both captures the vast majority of patients with ARDS and incentivizes application of LTVV early, when a failure to do so increases patient mortality (33).

The issue of how to define ARDS proved more difficult, highlighting the tension between feasibility and sensitivity in creating PM denominators. Initially, we contemplated a denominator based solely on International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) diagnosis codes, akin to NQF 0500, “Severe Sepsis and Septic Shock: Management Bundle” (34). We believed that this would simplify capturing patients with ARDS by avoiding additional manual data extraction and chart review. In addition, this group represents a patient population in which the clinician unmistakably recognized ARDS. However, there is ample evidence showing that many patients with ARDS are not recognized by clinicians (27, 35). Thus, an ICD-10-CM–based PM would have low sensitivity and limited impact in driving improvement.

Consequently, we considered an ARDS denominator definition with higher sensitivity, such as all patients who are mechanically ventilated because of acute hypoxemic respiratory failure. Workshop participants vigorously debated the practical merits of incentivizing LTVV indiscriminately applied to all patients with hypoxemic respiratory failure as a means of improving ARDS care. Ultimately, given recent data suggesting that LTVV may not benefit patients with respiratory failure without ARDS (31), as well as the requirement that ATS PMs remain evidence based, this approach was also rejected.

To improve ARDS specificity, we adopted a denominator similar to that of NQF 3215, “Adult Inpatient Risk Adjusted Sepsis Mortality” (36), which specifies identifying sepsis in administrative, billing, registry, and patient health record data. Identifying ARDS in this manner would require electronic health record (EHR) data extraction for the Berlin Definition criteria of ARDS: PaO/FiO, chest radiography reports, clinician notes seeking evidence of ARDS risk factors, echocardiogram, and other evidence of cardiac dysfunction (26). We recognize that this adds significant complexity, but the committee concluded that the current state of ARDS underrecognition requires it to support a meaningful PM.

Exclusions

Patients with chronic fibrotic lung disease (e.g., idiopathic pulmonary fibrosis and stage IV sarcoidosis) are excluded from the LTVV PM because their preexisting reduced lung compliance may render the LTVV inspiratory pressure goal unachievable. Although LTVV in this population may still provide benefit, it will not be measured under this PM. Patients who die within 12 hours of intubation are also excluded. Twice-daily ventilator settings should be collected and should count toward the PM until any of the following events: initiation of extracorporeal membrane oxygenation, a plan for withdrawal of life-sustaining treatment, meeting ventilator weaning criteria (37), or discontinuation of mechanical ventilation. Ventilator settings should not be reported from weaning assessments (e.g., pressure support trials), and patients subsequently deemed inappropriate for weaning are still included in the PM denominator pool.

Numerator

The PM numerator is the number of instances that LTVV was achieved, as defined by a tidal volume range of 4–8 ml/kg predicted body weight (PBW) and an inspiratory pressure less than 30 cm H2O. Participants considered whether the PM should adhere to a stricter tidal volume goal (i.e., 6 ml/kg PBW), because the primary evidence uses this goal, and the guideline itself recommends starting at 6 ml/kg PBW and increasing tidal volume to no more than 8 ml/kg PBW only in certain circumstances (16, 37). However, to simplify tidal volume reporting and mirror the recommendation language, the PM tidal volume was kept at 4–8 ml/kg PBW.

How to define LTVV adherence by the inspiratory pressure target also proved controversial. We allowed for the use of inspiratory peak pressures when plateau pressure is not routinely measured, but in order to incentivize the measurement of plateau pressure, we did not allow any variation in the pressure goal (both plateau and peak pressure goals are <30 cm H2O). Lowering inspiratory pressure—specifically, aiming for plateau pressure less than 30 cm H2O—is part of the guideline recommendation (16). However, in some subgroups, such as obese patients, an elevated inspiratory pressure may reflect reduced chest wall compliance rather than reduced lung compliance (38) and hence may incentivize inappropriately low tidal volumes and PEEP. Esophageal pressure (Pes) measurement may allow a more accurate assessment of the true transpulmonary pressure (39). However, routine use of Pes does not improve outcomes of patients with ARDS (40), and Pes and chest wall stiffness are not linearly correlated with body mass index (BMI) (39). Given the paucity of evidence to support a BMI threshold to exclude certain patients from the PM and the concern that subgrouping may exacerbate health disparities, there are no inspiratory pressure exclusions. As a general rule and in keeping with the requirement that ATS PMs remain evidence based, when insufficient evidence exists to support specific exclusions from the numerator or denominator, PM specifications should not deviate from the CPG recommendation.

Suggestions for scoring

As written, this LTVV PM may be applied either prospectively or retrospectively in the care of patients with ARDS. If done prospectively, we suggest screening for ARDS upon initiation of mechanical ventilation for acute hypoxemic respiratory failure (defined as PaO/FiO <300 or a comparable oxygen saturation as measured by pulse oximetry/FiO [41, 42]). Provided that no exclusions arise, each patient will have up to six data points over the 72-hour assessment period. Patients subsequently found not to have ARDS should be removed from the dataset. Adherence is defined as the proportion of data points, rather than patients, at an institution that are compliant with the PM. Given data demonstrating only 65% of mechanically ventilated patients with ARDS receive LTVV on Day 1 (27), we suggest a benchmark of 80% adherence. Although intended to incentivize an attempt at LTVV in 100% of patients with ARDS, a PM benchmark of less than 100% adherence acknowledges the group of patients in whom the LTVV goals are initially or never achievable for physiological reasons (e.g., severe acidosis, elevated intracranial pressure, and morbid obesity).

Target audience and pilot testing

This PM’s primary purpose is to improve patient-centered outcomes, specifically reduction of ARDS-related mortality by way of improving LTVV adherence. Because the PM relates to mechanical ventilation management, the target audience includes all healthcare professionals and health system leaders involved in the delivery of mechanical ventilation to patients with ARDS.

We identified several topics that require further research and attention during PM pilot testing. ARDS case identification remains challenging but is critical to the success of this PM. Thus, validation of ARDS case identification using ICD-10 codes as compared with EHR ARDS “sniffers” (43) is needed. Testing in sites where case identification will depend on manual chart review rather than automated EHR data extraction will better quantify the full spectrum of local resources that supporting this PM demands. Finally, the scoring system proposed is based on both current available ARDS literature and a desire to impose the least burdensome data-gathering requirements. Additional insights into the frequency at which ventilator settings are adjusted and recorded in clinical practice may inform future revisions to the scoring schema.

PM Implementation

Using the ATS’s strategy for guideline adoption (Figure 1), the third and final step is ensuring use of guideline tools in clinical sites. PM implementation is a resource- and time-intensive process. National endorsement of a PM, such as by the NQF, is one mechanism by which to achieve broad dissemination, but it is a multistep process that occurs over a period of months to allow for public comment and committee review (44). Before submission to the NQF, a PM must be operationalized, pilot tested, and revised to the point of suitability for NQF review. The planning and implementation of this phase may span 1 to several years (45). Davis’ hybridized Pathman-PRECEED (predisposing, reinforcing and enabling constructs in educational/ecological diagnosis and evaluation) model used in medical education (46, 47, 48) provides a useful framework for planning of implementation activities to optimize uptake or impact. Table 2 includes strategies and steps specific to pilot testing for the LTVV PM. An implementation evaluation framework, such as one that reports the acceptability, adoption, appropriateness, cost, feasibility, penetration, and sustainability (49) of the PM, will be used to evaluate pilot-testing cycles.

Table 2.

Pathman-PRECEED schema to implement low–tidal volume ventilation performance measure pilot testing

Types of Actions	Phase of Implementation
	Awareness	Agreement	Adoption	Adherence
Predisposing interventions (preparation)	Disseminate PM and intent to pilot test: • Educate ATS membership and ARDS stakeholders via published workshop report, ATS website, assembly meetings, direct outreach by content experts and thought leaders • Designate PM testing as an ongoing agenda item for QIIC meetings • Engage executive committee leadership in strategic planning, budgeting discussions	Solicit broad input, achieve consensus to proceed with PM pilot testing: • Provide a forum for stakeholders to comment on proposed PM and pilot testing • Encourage development of scholarship projects in parallel with pilot-testing process • Solicit participation by a broad range of ICU and hospital types to support broader early engagement for subsequent waves of implementing sites • Understand and address executive committee priorities	Formalize leadership committee, select testing sites, acquire resources: • Designate committee member roles, enlist support of senior director of implementation • Develop the package of information and education materials, including data collection process and database development • Define project timeline and milestones • Submit funding request	Confirm commitment of participants • Establish expectations regarding meeting frequency and timeline of project milestones • Receive funding, clarify funding renewal process
Enabling interventions (initiation)	Educate and train pilot-testing personnel, publicize program locally • Identify additional local stakeholders	Consolidate local support, address local concerns • Identify local program champions • Catalogue and respond to perceived and real barriers, facilitate a communication platform between pilot sites to share solutions to common issues • Enlist collaborators to demonstrate the value of the data for other purposes	Continually coordinate between ATS and pilot-testing sites, evaluate efforts • Ensure that sites have the right skills and capacity to implement the PMs. This will vary by context, such as different EHR systems. • Evaluate site processes, including assessment of resources required, identification of best practices	Use “feedforward” (50, 51) techniques to engage those involved at the site and unit levels • Maintain a database or registry that allows real-time unit-level access to allow immediate action in response to evolving data trends • Maintain a platform for intersite communication to share successes and lessons learned
Reinforcing interventions (sustainment)	Share site data locally • Create regular reporting processes or feed-in to existing quality dashboards	ATS and lead committee reiterate their commitment to complete specified pilot-testing period • Update ATS Executive Committee regarding progress toward NQF submission goal • Encourage and support completion of scholarship done in parallel with PM pilot testing	Codify and streamline best practices for data collection and reporting among sites • Embed processes into standard workflows and clinical duties	Evaluate suitability for broader PM adoption (such as NQF submission) • Review, evaluate, and report on PM pilot-testing experience using an established implementation evaluation framework • Design further pilot-testing iterations if needed vs. PM abandonment per pilot evaluation results and new scientific literature

Definition of abbreviations: ARDS = acute respiratory distress syndrome; ATS = American Thoracic Society; EHR = electronic health record; ICU = intensive care unit; NQF = National Quality Forum; PM = performance measure; PRECEED = predisposing, reinforcing and enabling constructs in educational/ecological diagnosis and evaluation; QIIC = Quality Improvement and Implementation Committee.

Challenges and Future Plans

Advancing the ARDS LTVV PM created by this workshop into its next phase of pilot testing will require dedicated funding, personnel, and coordination. The QIIC and the ATS Director of Implementation are the logical first choices to champion PM development and create PM pilot-testing teams. Ideally, establishing PM pilot-testing sites will yield mutually beneficial partnerships. The ATS will gain a network of diverse clinical settings that provide GI tool target users and future PM testing sites. Clinical sites stand to receive ATS expertise in support of local CPG implementation efforts as well as PMs and GI tools that better reflect and meet patient and stakeholder needs.

Both current and 2012 workshop participants endorsed the idea of developing and releasing PMs in parallel with CPGs to hasten GI and use the content expertise already assembled on guideline development committees. However, PMs necessarily follow CPGs because PMs cannot be developed until guideline recommendations and their strength are known. Furthermore, given the rigor and time that ATS CPG and PM development individually demand, guideline development committees may be overburdened by developing both. Finally, ATS is in the earliest stages of engaging in PM development; thus, the funding mechanisms to support and sustain PM development through pilot testing are not yet in place.

As a first step, we propose that the ATS issue “preliminary PMs.” As the development work of each new CPG nears its conclusion, a CPG “GI tool subcommittee” composed of QIIC-designated PM experts would identify appropriate candidate recommendations and write PM indicator statements and specifications. These preliminary PMs would be released concurrently with or soon after publication of the CPG. Preliminary PMs could be used as GI tools but with the disclaimer that as untested PMs, the ATS would not endorse their use in applications such as pay for performance. Clinicians and health systems could use preliminary PMs for individual improvement efforts, ideally feeding back their experience through scholarship or other channels. Then, on the basis of a combination of these voluntary PM user experiences, ATS membership input, and ATS Executive Committee priorities, a smaller group of preliminary PMs could be selected for full development, including formal pilot testing and NQF submission.

Conclusions

PMs are an important and underused component of the ATS’s GI tool portfolio. The ATS PM framework developed in 2012 was reviewed, endorsed, and applied to create an LTVV ARDS PM that will be shepherded through pilot testing by the QIIC with a goal of NQF submission. This process will provide an opportunity to cultivate internal expertise in PM development and inform future PM development endeavors, consistent with the ATS’s goal of patient and clinician advocacy by proactively engaging in the PM space.