A roadmap to operationalize and evaluate impact in a learning health system.

BACKGROUND: Many health systems invest in initiatives to accelerate translation of knowledge into practice. However, organizations lack guidance on how to develop and operationalize such Learning Health System (LHS) programs and evaluate their impact. Kaiser Permanente Washington (KPWA) launched our LHS program in June 2017 and developed a logic model as a foundation to evaluate the program's impact.
OBJECTIVE: To develop a roadmap for organizations that want to establish an LHS program, understand how LHS core components relate to one another when operationalized in practice, and evaluate and improve their progress.
METHODS: We conducted a narrative review on LHS models, key model components, and measurement approaches.
RESULTS: The KPWA LHS Logic Model provides a broad set of constructs relevant to LHS programs, depicts their relationship to LHS operations, harmonizes terms across models, and offers measurable operationalizations of each construct to guide other health systems. The model identifies essential LHS inputs, provides transparency into LHS activities, and defines key outcomes to evaluate LHS processes and impact. We provide reflections on the most helpful components of the model and identify areas that need further improvement using illustrative examples from deployment of the LHS model during the COVID-19 pandemic.
CONCLUSION: The KPWA LHS Logic Model is a starting point for future LHS implementation research and a practical guide for healthcare organizations that are building, operationalizing, and evaluating LHS initiatives.

INTRODUCTION

Integration of new research evidence into care delivery is often slow and inefficient. As a result, patients do not consistently receive high‐quality, evidence‐based care. Adopting the tenets of a Learning Health System (LHS) enables organizations to transform care delivery, improve patient health and experience, accelerate generation of patient‐centered knowledge, and reduce costs by decreasing the time for providers and health systems to ascertain and act on research evidence.

An LHS is a system in which internal data and experience are systematically integrated with external evidence, and that knowledge is seamlessly put into practice. Many organizations have explored and refined facets of an LHS, defining LHS researcher core competencies and core values, , articulating LHS priorities as the future of health services research, and exploring how to transform an organization into an LHS. Others have proposed frameworks for ethics and values to guide data collection and patient engagement in an LHS from a care delivery perspective. , , Methodologists have developed pragmatic clinical trials and robust, rapid‐cycle, non‐randomized methods for embedded research. Approaches to harmonize information technology infrastructure to facilitate access to and use of health systems data for learning are emerging across multiple settings. Publications describing LHSs reveal a complementary relationship between embedded pragmatic research and quality improvement (QI) activities.

Despite this progress, knowledge about developing, operationalizing, and evaluating the impact of LHS initiatives is scarce. While numerous papers list the desired constructs (fundamental components) of an LHS, details about how to operationalize those constructs, maximize the relationships among them, and assess their effectiveness and impact are lacking. , , A systematic review of LHS programs concluded that most work has been conceptual and unapplied, with “minimal focus on evaluating the impact.” Demonstrating the impact of an LHS could be a catalyst to higher acceptance and adoption of LHS concepts by health systems worldwide.

Building on decades of research partnership with care delivery, Kaiser Permanente Washington (KPWA) launched its LHS program in June 2017 as part of a strategic initiative to rapidly improve care and services for patients by integrating research capabilities such as evidence reviews, advanced analytics, program design and evaluation, and implementation science into strategic decision making. Group Health Cooperative was an early adopter in operationalizing the LHS and was acquired by KPWA in 2017. We created the LHS program to operationalize the LHS concept in our system, which is a single institution where the organization has control over many of the components of the LHS. This is distinct from broader LHS definitions that may work across multiple settings and institutions. The KPWA LHS program is led by an interdisciplinary team of operational, research, and medical staff at a research institute with 62 faculty and 246 staff. To address the lack of operational LHS models and to evaluate impact, KPWA LHS leadership reviewed published LHS conceptual models and measurement approaches. To achieve an evaluable depiction of an LHS, we constructed a logic model capturing the relationship between program resources, activities, and intended outcomes.

Here we present the KPWA LHS Logic Model, which provides a broad list of constructs relevant to LHS programs, depicts their relationship to LHS operations, harmonizes terms across models, and offers measurable operationalizations of each construct to guide other LHSs. The model identifies essential LHS inputs, provides transparency into LHS activities, and defines key outcomes to evaluate LHS processes and impact. Since the COVID ‐19 pandemic provided a unique opportunity to explore how rapid translation of science into practice occurs, we provide reflections on the most helpful components of the model and identify areas that need further improvement using examples from deployment of the KPWA LHS model during COVID‐19 to illustrate. Our purpose is to help organizations that want to establish an LHS program develop a roadmap for their organization, understand how LHS constructs relate to one another when operationalized in practice, and evaluate and improve their progress.

METHODS

We convened an interdisciplinary working group of LHS researchers with expertise in implementation science, QI, operations, communications, and translational research to identify core LHS constructs. We used a narrative review approach, which is appropriate when the question of interest is too broad or the body of evidence too sparse for a systematic review. One team member (KM) conducted a literature search to identify peer reviewed articles that included LHS models or listed LHS core constructs. We searched the PubMed and Embase databases, which have broad coverage of the academic literature. Search terms included variations of the base “learning health system,” including learning health system framework, learning health system model, learning health system program, and closely related synonyms. We did not include the term “translational research” or other related synonyms because these additional terms expanded the focus of the review beyond our intended scope. We also searched for common construct terms from existing LHS models (eg, “patient engagement,” “leadership”) and synonyms combined with the above terms and synonyms.

We scanned titles and abstracts of identified articles and read methods and background sections of studies that measured or attempted to measure LHS constructs. Since the field is new, we found widespread inconsistency in terminology. Therefore, we identified additional literature by following lead authors frequently cited in relevant articles. Ultimately, 17 articles met criteria for further review (Table 1).

TABLE 1

Citation list of models and frameworks analyzed to identify LHS constructs (listed chronologically)

	Model or framework	Citation
1	Physiology of a Learning System Model	Bohmer R. Designing care: aligning the nature and management of health care. Harvard Business Review Press 2009.
2	Rapid‐Learning Health Care System Model	Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207‐210
3	Requirements for a Learning Health System	Friedman C, Rigby M. Conceptualising and creating a global learning health system. Int J Med Inform. 2013;82(4):e63‐71.
4	Characteristics of a Continuously Learning Health Care System	McGinnis JM, Stuckhardt L, Saunders R, Smith M. Best care at lower cost: the path to continuously learning health care in America. National Academies Press; 2013.
5	Framework for a National‐Scale LHS	Bernstein JA, Friedman C, Jacobson P, Rubin JC. Ensuring public health's future in a national‐scale learning health system. Am J Prev Med. 2015;48(4):480‐487.
6	LHS Research Challenges and Questions	Friedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high‐functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43‐50.
7	LHS‐related IOM Reports	IOM Roundtable on Value & Science Driven Care. Integrating Research and Practice: Health System Leaders Working Toward High‐Value Care: Workshop Summary. National Academies Press; 2015.
8	Learning Health Care System Framework	Psek WA, Stametz RA, Bailey‐Davis LD, et al. Operationalizing the learning health care system in an integrated delivery system. EGEMS (Wash DC). 2015;3(1):1122.
9	Patient‐Centered Rapid Learning System Model	Wysham, N. G., Howie, L., Patel, K., Cameron, C. B., Samsa, G. P., Roe, L., … & Zaas, A. (2016). Development and Refinement of a Learning Health Systems Training Program. eGEMs, 4(1).
10	Learning Health Care System in the Veterans Health Administration	Atkins D, Kilbourne AM, Shulkin D. Moving From Discovery to System‐Wide Change: The Role of Research in a Learning Health Care System: Experience from Three Decades of Health Systems Research in the Veterans Health Administration. Annu Rev Public Health. 2017;38:467‐487.
11	Learning Health System Consensus Core Values	Friedman CP, Rubin JC, Sullivan KJ. Toward an Information Infrastructure for Global Health Improvement. Yearb Med Inform. 2017;26(1):16‐23.
12	Learn from Every Patient	Lowes LP, Noritz GH, Newmeyer A, et al. “Learn From Every Patient”: implementation and early results of a learning health system. Dev Med Child Neurol. 2017;59(2):183‐191.
13	Framework for Local and External Evidence Integration	Guise JM, Savitz LA, Friedman CP. Mind the Gap: Putting Evidence into Practice in the Era of Learning Health Systems. J Gen Intern Med. 2018;33(12):2237‐2239
14	Heimdall Framework for Supporting Characterisation of Learning Health Systems	McLachlan S, Potts HWW, Dube K, et al. The Heimdall Framework for Supporting Characterisation of Learning Health Systems. J Innov Health Inform. 2018;25(2):77‐87.
15	Conceptual Framework for Value‐Creating Learning Health Systems	Menear M, Blanchette MA, Demers‐Payette O, Roy D. A framework for value‐creating learning health systems. Health Res Policy Syst. 2019;17(1):79.
16	Care and Learn Model	Montori VM, Hargraves I, McNellis RJ, et al. The Care and Learn Model: a Practice and Research Model for Improving Healthcare Quality and Outcomes. J Gen Intern Med. 2019;34(1):154‐158.
17	Multilevel Framework	Harrison MI, Shortell SM. Multi‐level analysis of the learning health system: Integrating contributions from research on organizations and implementation. Learning Health Systems.e10226.

The team reviewed the papers and identified constructs based on strength of conceptual or empirical support for impact on an LHS, consistency in definitions, alignment with our own experience and potential for measurement. Psek and colleagues' framework from Geisinger Health Systems of nine LHS operational components emerged as the most comprehensive and applicable starting point to define core constructs. We compared the components to our experiences and the literature and found important omissions and a need for clearer operational definitions for several components. We used other studies to identify additional constructs, compared them across models, and clearly defined each in measurable terms. We reviewed the list of constructs and definitions and came to consensus on those central to an LHS. Our goals were to reduce redundancy, align constructs with salient domains (while acknowledging overlap and some fluidity), and ensure a comprehensive list of constructs that are key to a high‐functioning LHS. We combined constructs with different labels but overlapping definitions and parsed constructs that conflated underlying concepts.

We gathered examples of how each identified construct might be measured. We categorized each construct as a foundational LHS input, output or activity, or intended outcome. The resulting logic model portrays relationships among constructs and a framework to evaluate whether the LHS is achieving desired goals (Figure 1).

FIGURE 1

KPWA LHS Logic Model

RESULTS

The KPWA LHS Logic Model is based on the 17 frameworks and models identified in Table 1 and includes 24 constructs: 6 inputs, 9 outputs, and 9 outcomes. We describe each (Table 2) and provide examples of how to measure them to evaluate an LHS (Table 3).

TABLE 2

LHS logic model constructs with brief definitions

Topic	Description
I. INPUTS
A. People and partnership	Personnel and relationships involved in establishing and maintaining learning activities within and external to the organization 7
B. Health information infrastructure	Integrated and interoperable system that supports the data requirements of multiple stakeholders, digitally captures the care experience and allows real‐time access to knowledge for clinical care and learning 7
C. Prioritization	Process in which learning activities and opportunities are aligned with strategic goals across different levels of the organization 7
D. Funding	Mechanisms to fund the operational effort needed to enhance learning capability, as well as strategies for sustained funding of learning efforts 7
E. Improvement infrastructure	Leadership, policies and procedures to organize and facilitate improvement work 19
F. Ethics and oversight	Institutional guidance to navigate the differences, overlap, and similarities between quality improvement, clinical care, and research 7
II. OUTPUTS
A. Environmental scanning	Internal and external assessment of the current state of an issue or practice to identify gaps and recommend best practices
B. Evidence synthesis and translation	Summarize the academic literature for a clinical or research question and explain the application of existing evidence to the issue at hand 20
C. Data analytics	Inspect, cleanse, transform, visualize, and model data with the goal of discovering useful information, informing conclusions, and supporting decision‐making 21
D. Design	Design care based on evidence generated locally or elsewhere using pragmatic, timely, and flexible methods 7 , 17
E. Patient and family engagement	Integrate stakeholder values, experiences, and perspectives into LHS projects 22
F. Implementation support	Facilitate the process of putting to use or integrating interventions in the care delivery setting 22
G. Evaluation	Collect data and analyze results to show what does and does not work 17
H. Dissemination	Share results to improve care 17
I. Consultation	The provision of expert advice and counseling to inform decision‐making and promote learning 23
III. OUTCOMES
A. Knowledge‐to‐action Latency 24	The average time lag for clinical practices to adopt research evidence to improve care for patients
B. Systematic adoption of EBPs	Evidence of actual performance of a practice in the system and target impacts of that performance in practice 25
C. Systematic elimination of wasteful and ineffective practices	Reduction in clinical and operational practices that are cost‐ineffective or detrimental to health 26
D. Population health	Intermediate clinical health process and outcome measures for a population
E. Care experience	Patient satisfaction with care
F. Utilization/Cost of care 27	Utilization multiplied by the price of services, equipment, products, and prescription drugs
G. Work life for care teams 28 , 29	Clinical care and research team experience
H. Equity 30	Fairness in processes, outcomes, and relative costs
I. Programmatic return on investment	The cost of the LHS program investment over the outcomes achieved in learning, health, experience, equity, work life of teams, and costs of care achieved across the projects the LHS program supports

Abbreviations: EBP, evidence‐based practice; LHS, learning health system.

TABLE 3

Measurement of LHS logic model constructs

Topic	Level of analysis	Available measurement	Sample measures
I. INPUTS
A. People and partnerships	Organization or setting	Observation; Checklists	Do you have a team/department with dedicated time to meet to advance the program and solve problems? Does your team include members with diverse skills including expertise in implementation, research translation, quality improvement, and operations? Does your team have key stakeholder relationships in place to succeed?
B. Health information infrastructure	Organization or setting	Observation; Checklists	Does your organization have an EHR system? To what extent is the system used to input data? To what extent are there common data elements between information systems? To what extent does the EHR and data output interface with external systems like national population health registries? To what extent is population level data available at the point of care to care teams?
C. Prioritization	Organization or setting	Observation; Checklists	Is the LHS program directly cited in the organization's strategic plan? Indirectly? Is there documented alignment between LHS priorities and the organization's operating plan?
D. Funding	Organization or setting	Administrative data	To what extent does the organization provide analytic time, personnel, and resources to support LHS activities? To what extent is the LHS supported by federal, state, or local funding that is external to the organization?
E. Improvement infrastructure	Organization or setting	Administrative data	To what extent does your organization have leadership, dedicated staff time, policies, and procedures in place to facilitate improvement efforts?
F. Ethics and oversight	Organization or setting	Observation	To what extent do institutional guidelines and procedures exist to delineate quality improvement, clinical care and research? Is there a regulatory body to oversee risk management for quality improvement, clinical care and research projects?
II. OUTPUTS a
A. Environmental scanning	Organization or setting	Observation	Count of environmental scans produced
B. Evidence synthesis and translation	Organization or setting	Observation	Count of rapid literature reviews produced
C. Data analytics	Organization or setting	Observation	Count of reports that include original data analyses or descriptive data. Count of data models created that care delivery adopts
D. Design	Organization or setting	Administrative data	Count of stakeholder convenings Count of co‐design sessions Count of departments/providers/staff involved as partners
E. Patient and family engagement	Organization or setting	Administrative data	Count of patients and family members involved as partners Count of projects that included patient and family member input Documentation of how recommendations from stakeholders are applied
F. Implementation support	Organization or setting	Administrative data	Count of practice facilitator hours provided for care delivery initiatives Count of projects that required implementation support Count of resources and tools integrated into the EHR due to implementation support Documentation of changes made to implementation process including integration of an implementation framework, changes to workflows, support for clinical decision‐making, changes to implementation strategies, support pulling and applying data, and addressing context and barriers 31
G. Evaluation	Organization or setting	Administrative data	Count of evaluation reports completed Documentation of changes that occurred as a result of evaluation
H. Dissemination	Organization or setting	Observation	Count of internal and external publications, presentations, reports, and executive briefs Count of partnerships established with external organizations Documentation of LHS program's role in the internal spread of effective interventions
I. Consultation	Organization or setting	Administrative data	Count of consultation requests completed, categorized by type of request
III. OUTCOMES
A. Knowledge‐to‐action Latency 24	Organization or setting	Observation; Administrative data	Average time from publication of high‐quality evidence in academic literature to publication of an organizational guideline for a practice Average time from release of an EBP guideline to uptake among a percentage of the organization's providers
B. Systematic adoption of EBPs	Organization or setting	Observation; Administrative data	Count of new EBPs adopted by the organization Performance and impacts of existing EBPs at the organization
C. Systematic elimination of wasteful and ineffective practices	Organization or setting	Observation; Administrative data	Count of wasteful or ineffective practices reduced in the organization Performance and impacts of reduction of wasteful and ineffective practices
D. Population health	Individual consumer	Qualitative or semi‐structured interviews; Survey	Sample measure sets include HEDIS, UDS, NCQA
E. Care experience	Individual consumer	Qualitative or semi‐structured interviews; Survey	Sample measure sets include CAHPS, Press Ganey Patient retention
F. Utilization/Cost of care	Individual consumer	Administrative data	Sample measures including utilization multiplied by the price of services, equipment, products, and prescription drugs
G. Work life for care teams 29 , 32	Individual provider; Organization or setting	Survey; Qualitative or semi‐structured interviews; Focus‐groups	Sample quantitative measures including Maslach Burnout Inventory, internal provider and researcher satisfaction survey, Baldridge, Gallup Provider and staff retention
H. Equity	Organization or setting	Administrative data	Count of projects measuring outcomes by race, ethnicity, language, age, and other socioeconomic factors Count of projects using internal equity framework
I. Programmatic return on investment	Organization or setting	Administrative data	Cost of the LHS program investment over the outcomes achieved in learning, health, experience, equity, work life of teams, and costs of care Diversity of funding sources

Abbreviations: EBP, evidence‐based practice; EHR, electronic health record; HEDIS, Healthcare Effectiveness Data and Information Set; LHS, learning health system; NCQA, National Committee for Quality Assurance; Consumer Assessment of Healthcare Providers and Systems; UDS, Uniform Data System.

In addition to quantifying the number of deliverables for above categories, the quality of deliverables should also be assessed.

Documentation of changes made to implementation process including integration of an implementation framework, changes to workflows, support for clinical decision‐making, changes to implementation strategies, support pulling and applying data, and addressing context and barriers

KPWA LHS logic model constructs

Inputs

Inputs are essential elements for an organization to successfully operate an LHS. Inputs tend to be manifest variables, meaning they can be directly measured or observed. An organization can measure the extent to which inputs are present to determine readiness to transform into an LHS.

People and partnerships

These are the personnel and relationships involved in establishing and maintaining learning activities within and external to the organization. This construct measures well‐defined teams with diverse skillsets. Measurable elements include the existence of a team or department with dedicated time to meet to solve problems and support continuous learning; diversity in skills and backgrounds among team members, including representation from implementation science, quality improvement, clinical disciplines, operations, information technology, and analytics; and presence of key stakeholder relationships internal and external to the organization. Relationships occur at the patient, care team, clinic, department, and organization levels. Leadership is a critically important relationship to assess as they play a key role in setting expectations and communicating the value of learning.

Health information infrastructure

A key input for an LHS is an integrated, interoperable electronic health record (EHR) that supports the data requirements of multiple stakeholders, digitally captures care experiences and allows real‐time access to knowledge for clinical care, research, and learning. , EHR data also enables pragmatic trial designs and other uses of “real world data” to enhance the representativeness of research trials in an LHS, and allows for longer term follow‐up of treatment effects and harms at scale. Measurement of this construct includes the extent that the organization uses the EHR to input data, whether the EHR and other information systems have common data elements, the extent the EHR and data output interfaces with external systems like national population health registries, and whether population‐level data are available at the point of care. Given the integral role of data and analytics to an LHS, we separated the constructs of data infrastructure (input) and data analysis (output).

Prioritization

This is the process of intentionally aligning learning activities and opportunities with strategic and operational goals across organizational levels. At KPWA, this is a deliberate process that occurs annually and as needed if organizational strategic priorities undergo key shifts. Alignment with strategic goals is a foundational criterion we apply when selecting new LHS projects. This construct is measured by documentation of aligned priorities.

Funding

Financial resources are essential to developing an LHS. The KPWA LHS receives programmatic funding from the health system and external funding through research‐care delivery partnerships. Dedicated LHS resources are measured as total dollars committed by the organization plus total dollars committed by external sources.

Improvement infrastructure

The KPWA LHS found that an embedded QI infrastructure is a foundational LHS input. Leadership, policies and procedures to organize and facilitate improvement work across a system are essential for successfully piloting and spreading initiatives. The QI infrastructure must have an explicit, shared improvement methodology, approach, language, and culture; specific approach is not important. Lean, Six Sigma, and the Model for Improvement with Plan‐Do‐Study‐Act cycles are examples of healthcare improvement approaches.

Ethics and oversight

In an LHS, QI, clinical care and research may act synergistically, with QI and clinical care generating research, and research findings, in turn, informing clinical care and QI initiatives. Therefore, an LHS needs institutional guidance to navigate the differences, overlap, and similarities between these activities, including from Institutional Review Boards (IRBs) and compliance offices on assessing risks to participants while optimizing learning in any project on the continuum of QI, clinical care and research. Clear and streamlined IRB processes at KPWA make health system research nimble and efficient. This construct is measured as guidelines, procedures, and governance for LHS work.

Outputs

Outputs are key organization activities and deliverables that add value to care delivery. Measurement tends to be counts of deliverables. It is also important to assess the quality of these deliverables, as the quality of products is as important as quantifying production.

Environmental scanning

Internal and external assessments of the current state of an issue or practice help identify gaps and recommend best practices. Assessments can be a first step of a project or a distinct product. Environmental scans add value by identifying lessons learned, surfacing best practices, and connecting work siloed across organizational departments, other healthcare organizations, or related industries.

Evidence synthesis and translation

Health systems often struggle to interpret and apply existing evidence. Rapid literature reviews are a practical yet rigorous method to summarize evidence for a clinical topic and explain its application. , A vital part of this work is interpreting the strength of evidence so health system partners can make timely decisions about how and whether to proceed with adopting a practice.

Data analytics

Qualitative and quantitative data are integral to making informed decisions. At KPWA, we inspect, clean, transform, visualize, and model data to discover useful information, inform conclusions, and produce reports on progress toward prioritized strategic and operational goals to support decision‐making. Qualitative methods include writing and conducting surveys, interviews, and stakeholder focus groups. Quantitative methods include longitudinal and cross‐sectional analyses of clinical and business measures and advanced methods such as predictive models and machine learning. Measurement of analytic activities includes counts of reports that include data analyses and descriptive data and counts of analytic models the system adopts.

Design

To improve care, clinical teams, improvement advisors, informaticians, researchers and patients collaborate to design care based on evidence and stakeholder needs, using pragmatic, timely, and flexible methods. , At KPWA, design of care processes is most effective when the LHS program works with multiple stakeholders across departments to co‐create solutions for which operational teams are ultimately responsible. Human‐centered design and adult learning techniques help an LHS bridge siloed work and enhance engagement.

Patient and family engagement

Many models , , allude to integration of patient and family values, experiences, and perspectives into projects as a key learning and improvement activity that may also guide governance, decision‐making, and research activities. In addition to measuring the number of projects with patient involvement and the total number of patients included as partners, organizations should document how recommendations from patient and family stakeholders are applied so stakeholders know the value of their input. Closing the feedback loop for patient partners is an ethical imperative and central to sustained engagement.

Implementation support

Evidence‐based implementation approaches enable integration of effective interventions into clinical settings. Approaches include practice facilitation, clinical decision support, workflow development, EHR optimization, integration of implementation science frameworks and strategies, and tailoring interventions to organizational context. The KPWA LHS program creates change packages, conducts trainings, facilitates group learning sessions, anticipates and mitigates implementation barriers, and ensures relevant data are available, accessible, analyzable, and actionable to support implementation. Measurement of implementation efforts includes documenting products and processes created from implementation support, counts of practice facilitator hours, and numbers of implementation projects supported.

Evaluation

Collecting data and analyzing results to show what does and does not work is a key LHS program activity. An LHS program conducts evaluations and builds capacity in the health system to evaluate initiatives from the outset and describes initiative activities and their relationship to outcomes. While clinical leaders often initially request summative evaluations of outcomes, formative evaluations from the project start that clearly define the population of interest, core intervention components, or meaningful metrics are useful to provide teams with a priori benchmarks. We measure evaluation as counts of reports completed and documented changes in the organization or decisions that resulted from an evaluation.

Dissemination

Sharing results to improve care is a critical LHS output. Disseminating the processes and outcomes of LHS projects is crucial, as the field is young and the value of the LHS concept ill‐defined. An LHS may track all internal and external communications of its findings, including publications, presentations, reports, and executive briefs. This includes tracking communications to community partners and patients and families, which may include additional or alternative modes of dissemination. To maintain accountability for external dissemination, an LHS can also count external partnerships.

Consultation

Sharing expertise is part of continuous learning. The KPWA LHS provides a range of expert advice and counseling to inform decision‐making and promote learning within the KPWA delivery system. Consultations range from expert feedback on the content, methods, design, implementation, metrics, or evaluation of improvement or applied research projects, to operational advice about feasibility, budgeting, advocacy, staffing, and project management for an initiative. Consultation can be measured as counts of completed consults, categorized by type requested (eg, on metrics, implementation, operations).

Outcomes

Outcomes worth measuring include short‐term and long‐term goals, which may be measured at the patient, provider, organization or project level. The KPWA LHS program tracks process measures that are instrumental to our goal of rapidly implementing research evidence into care delivery, a core set of metrics based on the quadruple aim, health equity, , and programmatic return on investment. Project goals aligned with the strategic and operational plan determine a project's specific metrics.

Knowledge‐to‐action latency

Reducing the lag time for clinical care teams to adopt new research evidence to improve patient care is the central focus of an LHS. We measure this by monitoring average time from publication of high‐quality evidence (eg, Cochrane reviews, meta‐analyses) to publication of an organizational guideline for a practice, and average time from release of an internal evidence‐based practice (EBP) guideline to uptake among a percentage of the organization's care teams.

Systematic adoption of EBPs

This outcome is the evidence of actual EBP performance in the system and the target impacts of that performance in practice. We measure this as counts of new EBPs adopted by the organization, and by tracking the performance and impacts of existing EBPs.

Systematic elimination of wasteful and ineffective practices

Reducing clinical and operational practices that are cost‐ineffective or detrimental to health is the inverse of helping the organization quickly adopt EBPs. The LHS aims to improve outcomes and focus resources according to organizational priorities. We measure this complex outcome as counts of wasteful or ineffective practices reduced and by tracking the performance and impacts of these reductions drawing upon health economics. , ,

Population health

This is part of the quadruple aim, defined as process and health outcome measures for a population. Our LHS program monitors patient‐level clinical health outcomes for almost all projects. In addition to identifying and measuring distinct patient‐level clinical measures, we also use a composite measure that looks at evidence‐based care gaps across a patient population including both process and outcome measures. The goal of an LHS intervention is to improve individual outcomes and health system functioning to assure quality across measures. We find it most efficient to align health measures with national quality dataset definitions such as from the Healthcare Effectiveness Data and Information Set (HEDIS), the National Committee for Quality Assurance (NCQA), the National Quality Forum (NQF), and an organization's priority clinical quality metrics.

Care experience

An LHS can measure patient experience using validated research instruments including items from the Patient‐Reported Outcomes Measurement Information System (PROMIS), operational surveys such as the Consumer Assessment of Healthcare Providers and Systems (CAHPS) and from qualitative feedback directly from patients and families. , Long‐term experience outcomes include voluntary disenrollment rates and patient retention over time.

Utilization/Cost of care

Examining programmatic impacts on care costs along with impacts on quality and experience ensures a balanced approach to understanding the value of interventions. Ultimately, the goal of the LHS is advancing resource stewardship to understand the relative impacts on intervention costs. However, many health system interventions are not designed to impact price. Thus, evaluating impacts on care utilization may be a reasonable proxy for cost reductions. Especially of interest are interventions to reduce inappropriate or unnecessary use of high‐cost services.

Work life for care teams

The work experience of clinical care and research teams is a core metric for most KPWA LHS projects. Sample measures include the Maslach Burnout Inventory, metrics from the Primary Care Team Dynamics Survey, the Dimensions of Learning Organization Questionnaire, internal clinical and research team satisfaction surveys, and care team interviews. , , Provider and staff retention are longer‐term outcomes. ,

Equity

Improving system capacity to capture and use data on social needs and demographics is integral LHS work. Defined here as fairness in processes, outcomes, and relative costs, the KPWA LHS program stratifies quality outcomes by race, ethnicity, language, age, and other socioeconomic factors wherever possible and designs projects using an equity framework and an internal equity, inclusion, and diversity consult service.

Programmatic return on investment

This is an aspirational measure that looks at the cost of the LHS program investment over the outcomes achieved in learning, health, experience, equity, work life of teams, and costs of care across the projects the LHS program supports. We also track diversity of funding sources as a standalone financial health metric to measure the amount of new funding brought into the organization by leveraging the current investment.

DISCUSSION

Building on more than 13 years of experimentation since the Institute of Medicine introduced the LHS concept, and more than 17 LHS frameworks, we present a pragmatic, generalizable logic model that describes the inputs, activities, and outcomes of the KPWA LHS program. We share sample measures and measurement strategies for each model construct, for ease of evaluation. This model embraces, consolidates, and synthesizes constructs from other publications to create a roadmap for organizations embarking on the LHS journey, in hopes of harmonizing language to support future knowledge accumulation. Below, we present lessons learned from our experience as an LHS and highlight effective strategies and areas for improvement. The COVID‐19 pandemic provided an opportunity to test our LHS program's infrastructure and capabilities. Thus, we illustrate with examples from actions taken in response to the pandemic.

Effective strategies

Central to the acceleration of learning at KPWA is LHS program sponsorship (Funding). This program, seated in research, works closely with our QI department and leaders in medicine, operations, innovation, and strategy to leverage research capabilities to improve care (People and Partnerships). Having an LHS with real world data generates practice‐based evidence, which can identify evidence‐based clinical programs that naturally occur within the system and increase scalability. The program serves in part as a switchboard, enabling the research group to plug in when interesting natural experiments arise (Prioritization) and for care delivery to better understand and use the resources of the research team. While the research institute engaged in traditional science including the first trial of a COVID‐19 vaccine, our care delivery partners, and regional and state health authorities leveraged our LHS program to inform real‐time decision‐making through evidence implementation and knowledge translation.

For example, at the start of the COVID‐19 pandemic, the LHS program provided rapid literature reviews on telehealth best practices (Evidence Synthesis and Translation) and produced predictive analytic models to identify members at highest risk of COVID‐19 complications, suicide attempt, or care delays to received targeted outreach calls (Data Analytics). This work was based on 3 years of development and pilot work by our advanced analytics team. This team also supported development of models for hospital and intensive care unit utilization for the state at the request of the Washington governor's office (Consultation). The LHS now conducts biweekly rapid literature review and synthesis for health system leaders on notable findings on COVID‐19 treatments and vaccines (Evidence Synthesis and Translation).

We also assisted with the health system's move to virtual care with a focus on equity. KPWA responded to the pandemic by designing new ways of caring for certain clinical conditions virtually based on LHS rapid literature reviews and patient and provider data from related evaluations. Widespread protests against systemic racism called attention to health disparities and highlighted the need for virtual care to be equitable and to mitigate health disparities. We leveraged existing capabilities to rapidly develop a mixed‐methods approach to virtual care design that includes dashboards, surveys, focus groups and interviews to measure and track quality and patient experience, including a focus on key populations defined by race, ethnicity and language (Evaluation). Additionally, the LHS program used patient feedback to inform decisions in shifting 90% of care to telehealth. We could collect these real‐time patient data because the KPWA LHS regularly conducts mixed‐methods evaluations that include formative evaluation experts and traditional researchers to provide real‐time feedback and strong summative findings (Evaluation). Often, administrative data are insufficient to inform implementation, so the LHS invested in elevating patient voices through focus groups, interviews, and surveys.

Finally, the KPWA LHS developed clear processes for ethical oversight of activities (Ethics and Oversight). Early in the KPWA LHS, program leadership worked with the KPWA IRB on an application form (available upon request) specific to LHS projects that provides critical information the IRB needs to make determinations. The LHS and IRB agreed to a common set of ethical considerations, enabling rapid review of COVID‐19 modeling projects.

Established partnerships among researchers, care teams and health system leaders made these efforts possible (People and Partnerships). Specifically, the LHS program proactively scans for windows of opportunity where evidence and data can influence organizational decisions. These windows, where true equipoise on decisions exists, can be vanishingly brief. In the case of the COVID‐19 pandemic, we leveraged previous work to understand the content expertise and disposition of our organization's scientists and staff, enabling us to match them to decision makers working on questions of telehealth, modeling, epidemiology, and vaccines (Consultation).

Areas for improvement

The COVID‐19 pandemic also exposed areas for improvement for our LHS program. At the outset of the crisis, we attempted to identify and share resources with stressed community health centers to help them stay abreast of ever‐changing guidelines and protocols (Dissemination). We found that we did not have the organizational infrastructure or relationships in place to prioritize and translate the resources we had to community‐based settings (People and Partnerships). We need to build pathways to more quickly and efficiently adapt and disseminate our resources to reach a broader population. Finally, we did not have patient and family partners ready to participate in co‐design and provide feedback for our organization's rapid switch to virtual care (Patient and Family Engagement). We responded by establishing a diverse group of patients and family members to work alongside the LHS to develop and prioritize ongoing and forthcoming initiatives.

Limitations

The narrative review approach may be subject to bias. Our logic model is not a comprehensive list of measures or measurement strategies, but rather provides a starting place for organizations to begin evaluating the structure and value of their LHS program. The logic model does not include contextual factors such as learning climate and culture. Rather, our model focuses on critical elements central to an LHS's functioning and could be used in complement with an ecologically or organizationally focused model, such as the multilevel framework of factors influencing organizational learning. Finally, given our program's state of development we do not yet have outcome measures.

Conclusion

The KPWA LHS logic model identifies critical inputs for developing a successful LHS, ensuring transparency of its activities, and defining key outcomes to evaluate impact. Organizations can use this model as a starting point to build, operationalize, and evaluate their LHS efforts. Next steps will be empirically testing this model in other settings and refining it based on findings. Further work may include conducting a process to assess and validate the standardized structure and terminology we offer to guide the LHS field. Future research should evaluate the proposed process and outcomes measures of LHSs over time to determine the value of embracing continuous learning in healthcare organizations.

CONFLICT OF INTEREST

The authors have no conflict of interest regarding this research.