Seeing the quality improvement forest through the quality improvement trees: A meta-synthesis of case studies in Florida and Georgia.

OBJECTIVES: To identify important characteristics of quality improvement applications for population health and healthcare settings and to explore the use of quality improvement as a model for implementing and disseminating evidence-based or best practices.
METHODS: A meta-synthesis was used to examine published quality improvement case studies. A total of 10 published studies that were conducted in Florida and Georgia were examined and synthesized using meta-synthesis (a qualitative research methodology) for meaningful insights and lessons learned using defined meta-synthesis inclusion criteria. The primary focus of the analysis and synthesis were the reported processes and findings that included responses to structured questioning in addition to emergent results from direct observation and semi-structured open-ended interviewing.
RESULTS: The key insights for the use of quality improvement in public health and healthcare settings included (1) the essential importance of data monitoring, analysis, and data-based decision making; (2) the need to focus on internal mutable factors within organizations; (3) the critical role of quality improvement team group dynamics; (4) the value of using a quality improvement collaborative or multi-clinic quality council/committee for sharing and comparing performance on key metrics; and (5) the need to identify a quality improvement approach and methods for clarification as a structured quality improvement intervention.
CONCLUSION: In addition to the advantages of using quality improvement to enhance or improve healthcare and public health services, there is also potential for quality improvement to serve as a model for enhancing the adoption of evidence-based practices within the context of dissemination and implementation research.

Introduction

Quality improvement (QI) has become a priority for both healthcare and public health.[1-4] The extent of institutionalization of QI in medicine can be seen in the Accreditation Standards for Graduate Medical Education (ACGME) Common Standards which specify requirements for all graduate programs in Quality Improvement (VI.A.1.b) including the following: Education in Quality Improvement (VI.A.1.b.(1)), use of Quality Metrics (VI.A.1.b.(2)), and Engagement in Quality Improvement Activities (VI.A.1.b.(3)).[5] The Institutionalization of QI in public health can be seen in the requirements for public health agency accreditation that includes a Domain 9 for QUALITY IMPROVMENT that specifies the following: Use a Performance Management System to Monitor Achievement of Organizational Objectives (Standard 9.1) and Develop and Implement Quality Improvement Processes Integrated into Organizational Practice, Programs, Processes, and Interventions (Standard 9.2).

Use of QI in healthcare and public health can be challenged by interpretations of QI that range from (1) using the term for any and all efforts to improve quality, however unstructured, to (2) using as many QI techniques as possible without clarity of the big picture, or lack of purpose related to “Big QI” or QI culture.[2,6-9] There are text books full of QI techniques, all potentially useful.[10,11] Applied indiscriminately without connection to the big picture, the extensive use of methods might actually undermine the purpose of QI. However, calling every effort to improve quality, without using any QI principles or techniques, is contrary to the basic foundations of management science and industrial/operant engineering that gave birth to QI, whose origin is frequently attributed to W.E. Deming.[12]

Research on QI for healthcare has increasingly focused on QI’s importance for Implementation Science, but issues of what are the most useful methods of QI for healthcare are not well established.[13] To help provide insights into critical QI methods, we examine the experiences from applying QI within the context of the QI culture, particularly as those elements support the main purpose of QI, and summarize lessons learned. The purpose is “interpretive explanation, not predictive.”[14] Addressing the problem of clarifying essential characteristics of QI within the context of a plethora of QI approaches and techniques, the purpose of this meta-synthesis is to abstract the lessons learned from selected QI evaluations within the context of QI culture to guide implementation of QI activities within public health and healthcare. In particular, we discuss these findings for their implications in using QI in these settings and as a tool for public health and healthcare implementation research (IR).

Methods

This study utilized meta–synthesis, a qualitative research design that is distinguished from systematic analysis and meta-analysis. The purpose of meta-synthesis is gaining insight and interpretation across case studies rather than generalization that is more associated with systematic analysis and meta-analysis. The sources of the data are mixed-method case studies in contrast to meta-analysis of quantitative data of studies with similar characteristics or attempts to review the broad universe using predetermined criteria for exclusion and inclusion associated with systematic reviews of the literature. In particular, meta-synthesis has an interpretive, rather than aggregating intent.[15]

The meta-synthesis methods used to synthesize the QI studies reported in the selected literature are based on an approach derived from meta-ethnography.[16-19] This approach has been utilized to study a wide variety of health issues such as diabetes control, depression treatment, age-related macular degeneration, brain stimulation and Parkinson’s, the physical environment and exercise, and impact of fatigue on long-term conditions.[18,20-24] As the use of meta-synthesis has dramatically increased since 2000, the methodological issues involved in meta-synthesis have also received increasing attention.[25] Doyle’s[14] comparison of meta-synthesis, meta-analysis, and systematic literature review may be particularly useful in distinguishing the differences in purposes, sources of data, data collection, processes, and products. Of particular relevance to the methods used for this synthesis, meta-synthesis uses purposeful sampling in contrast to random sampling and exhaustive reviews.

Although Lee et al.[25] discuss a wide range of different methods used by different authors within the context of a seven-step meta-synthesis process, they maintain that “Noblit and Hare’s seven-step iterative process remains the primary organising (sic) device for conducting meta-ethnography.”[24] Using those seven steps, described in Table 1, the first two methodological steps of the meta-synthesis were (1) clarifying the purpose (previously described in section “Introduction”) and (2) determining which studies (the sampling approach) would be analyzed. The focus of this synthesis involved purposeful sampling, consistent with meta-ethnography, based on a group of QI case studies that included both quantitative and qualitative (mixed-method) data/information. Key criteria for inclusion of published studies were as follows: (1) QI was a major component of each study and (2) overlap was evident among investigators (a criterion for meta-synthesis), with at least one investigator common to all case studies.[14] This purposeful sampling approach to case studies enabled the selection of health-related QI case studies that involved reported changes in public health and healthcare organizational behavior/culture, which other systematic searches (i.e. PubMed) did not yield. The overlap among investigators also resulted in a geographic focus (Florida and Georgia) as well as a methodological consistency reflecting community- and/or practice-based research and evaluation.

Table 1.

Steps in meta-synthesis based on Noblit and Hare’s seven steps.

1. Clarified the purpose of the meta-synthesis

2. Delineated the studies that would be the focus of the meta-synthesis

3. Examined (read) the reported studies for metaphors or themes

4. Determined the relationship of the metaphors/themes to QI culture

5. Translated/derived critical if not essential concepts for QI culture

6. Synthesized the key concepts as overarching lessons learned

7. Summarized the key concepts for practical implementation of QI in healthcare and public health settings

QI: quality improvement.

Steps 3–7, which methodologically are most aligned with the methods of analysis and interpretation in research parlance, were the primary focus of Lee et al.’s[25] discussion of methodological issues. Most critical for meta-synthesis is “intensive, repetitive, and above all highly active reading.”[24] More specifically, we used methods of content analysis including (1) initial review of studies for key constructs/themes/variables, (2) development of unique codes for each key theme, (3) reexamination of each case study, applying the codes to the themes as they appear in each of the case studies, and (4) development of consensus on critical elements of QI culture. As key themes were identified and placed in a matrix on a vertical axis, each of the studies, identified on a horizontal axis of the matrix, was then reexamined for the identified themes. The development of the matrix involved five iterations wherein the themes were added, refined, and consolidated as the studies were reexamined. The later rounds included computer-assisted searches, primarily using the Adobe search function, as all of the files were in PDF format. The use of the search function was conducted using one or more words associated with a theme followed by examination of the context of how the word was used to assess its relationship to the theme.

Results

The reports and studies examined through the use of meta-synthesis for this study are briefly displayed in Table 2. A total of 10 published studies were examined for their implications for using QI in public health and healthcare and using QI as a tool for public health and healthcare IR. They include a QI immunization evaluation project in a large urban area of Northeast Florida, evaluation of QI efforts within Georgia public health agencies, evaluation of community asthma projects utilizing QI in Georgia, and sexually transmitted disease (STD) quality and cost effectiveness efforts in Florida county health departments (CHD).[26-35]

Table 2.

Published studies examined for this analysis.

Reference	Journal	Geographic location	Setting (public health, healthcare, community)	Primary health service focus(es)	Primary outcome/performance measure(s)	Time period
Livingood et al.[26]	AJPM	North Florida (Jacksonville)	County public health agency /multiple clinics	Delivery of childhood immunizations	Immunization rates for 2-year-old children	2009–2011
Livingood et al.[27]	BMC HSR	Georgia	Multiple CHD districts	General public health agency services	QI culture	2013–2014
Livingood et al.[28]	JPHMP	Georgia	Multiple CHD districts	General public health agency services	QI culture	2010–2011
Livingood et al.[29]	Frontiers PHSSR	Georgia	Multiple CHD districts	General public health agency services	Clinic wait times; HIV testing and counseling	2011–2012
Alexander et al.[30]	Frontiers PHSSR	Central Georgia	Single district in single CHD	Teen reproductive health services	Wait times for teen clinic	2012–2013
Marshall et al.[31]	Frontiers PHSSR	Central Georgia	Single district with multiple CHD	HIV services	HIV reporting for screening, testing, etc.	2012–2013
Woodhouse et al.[32]	Hlth Prom Prac	Georgia	Diverse community settings (hospital, public health, school, university based)	Asthma services	Asthma plans, ED visits, hospitalization, use of medication, and trigger removal	2008–2013
Woodhouse et al.[33]	Pop Hlth Mng	Georgia	Diverse community settings (hospital, public health, school, university based)	Asthma services	Asthma plans, ED visits, hospitalization, use of medication, and trigger removal	2008–2014
Livingood et al.[34]	Frontiers PHSSR	Florida	Public health agencies across state	STI services	Reduce costs; decreased time to treatment	2014–2016
Bilello et al.[35]	JPHMP	Florida	Public health agencies across state	STI Services	Decreased time to treatment	2014–2016

QI: quality improvement; ED: emergency department; STI: sexually transmitted infection; CHD: county health department.

Data that were analyzed came from both structure inquiries as well as direct observation and input provided without structured prompts.

Structured inquiries

Consistently recurring characteristics of QI culture included QI team functioning/group dynamics, organization support, and tracking and use of data. These themes emerged in no small part due to questions or prompts by investigators using items derived from other research related to quality improvement collaboratives[36] which included Likert-type responses from strongly agree to strongly disagree. These investigator-initiated prompts are summarized in Table 3.

Table 3.

Content of investigator-initiated prompts used in many of the reviewed studies.

1. QI team functioning/group dynamics

Members of my unit were directly involved in making changes for QI

QI team participation enhances collaboration across job responsibilities

Members of my unit are motivated to implement changes for QI

My unit considers continuous improvement as part of working process

Information, ideas, and suggestions are actively exchanged for quality improvement

2. Organization support

My unit supports goals and activities

Management prioritizes success of QI

3. Tracking and use of data

Goals are readily measurable

My unit uses measurements to plan changes

My unit tracks progress continuously

QI: quality improvement.

Emergent results

Considerable information emerged where there were not structured inquiries. This included information on specific QI techniques, the QI approach that was used, and other factors that were not consistent but did emerge in some specific settings such as perceived barriers to QI.

Related to each source of data (published study), a summary of results related to key elements of QI culture including QI culture characteristics, cross-site sharing of data, and major conclusions/lessons learned for QI are described in Table 4.

Table 4.

Key elements of QI culture.

Reference	Sources of data on culture	QI culture characteristics	Cross-site sharing of data	Major conclusions/lessons learned for QI
Livingood et al.[26]	Direct observation, interviews, and survey	Used PDSA model of QIUse of QI teamManagement support for QIStaff involvement in change/staff motivationMeasurable activities/outcomes/use of benchmarksUses measurement to planStaff consider QI basic part of workMonitors and tracks progressActive exchange of ideas/informationMulti-disciplinary teamworkOrganizational developmentAgency culture changeQI merged with evaluation/mixed methodAgency culture changeSpecific QI techniques	Scoreboard/dash-board displayed across all clinics	Initial aversion to QI, little perceived value.QI, combined with evaluation, supports the refinement of QI approach.QI improved population health outcomes and improved QI agency culture.
Livingood et al.[27]	QI culture survey	District and management support for QIStaff motivation, a key element of QIMeasurable activities/outcomesUses measurement to planStaff consider QI basic part of workMonitors and tracks progressActive exchange of ideas/information	Not reported (although monthly PBRN meetings)	QI culture different than simply counting the number of QI methods.Public health PBRN supportedQI interventions enhance agency QI organizational culture.
Livingood et al.[28]	Closed-ended QI culture survey items and open-ended questions	Organizational support for QI (8 items)Multi-disciplinary teamwork (14 items)Use of measures including setting targets and collecting and monitoring data (12 items)Collaborative structured processes (10 items)	Not reported (albeit monthly PBRN meetings)	Districts as multi-jurisdictional entities can support agency QI.
Livingood et al.[29]	Historical/archival review	QI as development process for organizationsTraining in QIUsed PDSA model of QIPerformance measure for outcomes and qualitative observations for use of QI processes	Not reported (albeit monthly PBRN meetings)	Although each agency selected different projects, common QI techniques were used across projects.
Alexander et al.[30]	Direct observation and interviews	Multi-disciplinary teamUsed PDSA model of QI and structured QI process (PDSA cycle, process map, root-cause analysis)Data monitoring—control chartsCommunication across teamLack of follow-up and commitment by some	Control chart displayed	Effective use of data to inform decision making was a slow culture change process.The QI team members started to make progress after QI team started monitoring data.
Marshall et al.[31]	Direct observation and interviews	Used PDSA model of QIBroadly representative consortium teamBroad delegation of assignmentsQI experienced leader/championAdministrative supportConsistent and frequent use of dataData-driven decisions	Displayed quarterly at local consortium meetings	Importance of a dedicated champion for any QI effort.Consistent and frequent use of data to identify ways to improve outcomes.Using data to drive decisions fosters a QI environment.Importance of support from various leadership levels.
Woodhouse et al.[32]	Direct observation, interviews, and archival data	QI merged with evaluationUsed PDSA model of QIUsed logic models extensively to clarify process and outcomes measures for PDSAMeasurable activities/outcomesPerformance measure driven	Not reported	Commonalities between QI and health promotion–based evaluation, that is, focus on both outcome and process measures.Difference between QI and health promotion–based evaluation, that is, origins and roles in using data.
Woodhouse et al.[33]	Direct observation, interviews, and archival data	Great variation in settings and programsCommon outcome measures were a challengeUsed benchmarksNeed to adapt to each unique organization	Not reported (regular meetings of local program leaders reviewing status)	Program development with developmental approach to evaluation may be needed.
Livingood et al.[34]	Direct observation, interviews, archival data, and surveys	Multi-site QI collaborativeUsed PDSA model of QIAttention and reporting of QI processesUse of QI teamsQI experienced leader/championMeasurable activities/outcomesMonitors and tracks progressData-informed decision making	QI collaborative monthly report	QI-based implementation research can adapt the best practice to the context of each unique CHD rather than create new knowledge. QI approaches may be an important tool to assist in the adaptation of new technologies.
Bilello et al.[35]	Direct observation, interviews, archival data, and surveys	Multi-site QI collaborativeUse of QI teamsMeasurable activities/outcomesMonitors and tracks progressPerformance data driven, discussed regularlyUse of QI to adapt to barriers and emerging challenges	QI collaborative monthly report	QI approaches can be effective in increasing important outcome measures.

QI: quality improvement; PDSA: Plan-Do-Study-Act; PBRN: practice-based research network; CHD: county health department.

Many of the themes were recurring across all of the studies. Themes that occurred in at least 50% of the studies included the following:

Measurable activities/outcomes (nine studies). Measurable activities/outcomes were core components of the QI interventions studied. The use of benchmarks was also included under this theme, although this did not occur as frequently.

Use of QI team (eight studies). Other related constructs were also included under this theme including staff ownership, QI being a basic part of work, and staff engagement in change, although these points were not as frequently mentioned.

Monitors and tracks progress (eight studies). Related to having measurable outcomes and processes, tracking progress was common across the studies, but is an important distinction because it indicates use of the data.

QI merged with evaluation/mixed-method evaluation (eight studies). QI itself is not a research or evaluation design, but combined with evaluation designs does enable reporting of results and lessons learned. A mixed-method design was frequently reported and related constructs such as qualitative observation, reporting of QI process, and use of logic models were also identified in the studies, albeit less frequently the overarching theme.

Uses data to plan/make decisions (seven studies). Providing further evidence of using data, this theme was also recurring.

Used Plan-Do-Study-Act (PDSA) model of QI (seven studies). The model for improvement, commonly known as the PDSA approach to QI, was almost exclusively used as the QI model.[11]

Other frequently occurring themes included multi-disciplinary teamwork (five studies), organization/staff/community development (five studies), management support for QI (four studies), experienced QI leader/champion (four studies), great variation in settings or programs and need to adapt to each setting (four studies), specific QI techniques (four studies), training in QI, multi-site collaboration (three studies), and practice-based research (three studies). Other themes that were identified occurred only once or twice.

Viewing the analysis beyond repetition, or the number of studies in which a theme appeared, the robustness or focus of the narrative also provided important insights. What might be considered underlying or implied themes, although not explicitly stated, also had potential for critical insights. These issues emerged through reexamining the studies for a major focus or an implied theme beyond repetition of specifically state idea and include the following:

QI model/approach. A QI approach or model was typically identified. The model for improvement, or PDSA model, was the predominant model used for QI.[11] Despite training in QI methods for staff/graduate student support for Georgia projects that was primarily accomplished through the Six Sigma model, which has substantial emphasis in QI methods, specific QI methods did not appear essential.[10] It should be noted that PDSA, a preferred model by the Florida Department of Health, is strongly encouraged for use by all Florida CHD.[11] The Study-Act phases of the PDSA model also have an emphasis on reviewing data in light of the planned changes and acting on the results of that data review, which complements one of the previously described structured inquiries, tracking, and use of data.

Specific QI techniques. Specific QI methods such as root-cause analysis, control charts, or Pareto charts were reported but not consistently mentioned. Although tracking and displaying data was consistently reported, different methods appeared to be used, ranging from more QI-aligned control charts to use of emoticons to show success in meeting objectives or lack of success.

Barriers. The Northeast Florida Immunization study explicitly revealed that QI can be viewed negatively by staff, particularly when it is simply seen as more top-down impositions on staff with little or no perceived value by the staff.[26] Barriers within the Georgia projects included a lack of QI culture (no history of organizational involvement with QI), lack of timely access to data on the key performance measures, lack of support for QI by other employees, and lack of follow-up to QI team meetings. On the other hand, the Georgia projects also indicated that previous experience with organizational involvement with QI teams and use of data for decision making appeared to facilitate successful QI efforts.

Mutable internal organization factors. A root-cause analysis process for the immunization project in the Florida CHD initially focused on a number of factors outside the organization such as the behaviors of clients or the lack of support by upper-level or state management. QI evaluation feedback enabled the county department to refocus the QI efforts on mutable factors within the clinics, resulting in a number of QI PDSA initiatives that were accompanied by substantial progress in improving the immunization rates. Efforts in Georgia and the Florida STD QI projects focused on mutable factors within those organizations, sometimes involving considerable participatory problem-solving approaches to identify the opportunities for improvement.

QI collaborative or council. All of the studies involved some form of QI collaborative or enterprise-level council. The Florida CHD Immunization QI Project involved an enterprise-level quality council and a cross-site evaluation team where sharing and cross-site problem solving were frequently used. The Georgia public health initiatives involved a public health practice–based network that met monthly to share and discuss issues. The Georgia Asthma Project involved a multi-site participatory evaluation team where barriers and success were discussed. The Florida STD Project also involved a public health practice–based network that utilized multi-county problem sharing. Having opportunities to learn from others about QI practices from a number of QI teams appeared to be an important asset for advancing quality.[33]

Discussion/conclusion/implications

Potential value to QI in healthcare and public health

The insights developed from this meta-synthesis have potential value to both the application of QI in healthcare and public health as well as value for IR. Although QI, with its foundations in management science and industrial engineering, is typically not included in social science or health research, this qualitative research (meta-synthesis) on QI yields important insights that can inform the use of QI for intervention and implementation research for healthcare and population health. Five insights or lessons learned are particularly worth highlighting:

Data monitoring, analysis and data based decision making. Some efforts to assess QI culture tend to omit the importance, or at least give minimal focus, to monitoring data and displaying and using data for decision making.[37] Public health agencies and healthcare providers collect and report extensive amounts of data, but frequently do not use the data to inform their own decision making, particularly related to enhancing the quality of their services. The Study-Act part of PDSA and the Measure-Analyze part of Six Sigma’s DMAIC make data analysis and monitoring an essential part of the QI process. Some type of display of tracked results would appear to be essential. Mechanisms for displaying progress may vary extensively such as control charts that are more specifically identified as QI methods, dashboards, or smiley/frown emoticons (accompanied by data) showing successes or lack thereof. Some form of feedback based on data appears to be necessary to inform QI teams and reinforce ownership and action by the QI team, essential characteristics of QI.

Focus on internal mutable factors within organization. Brainstorming and root-cause analysis can yield a plethora of causes which can be used to avoid taking ownership of QI. This may be obvious, but engaging staff in brainstorming and root-cause analysis can shift the focus from internal change for improved performance to blaming external factors out of their control. Having a clear focus on mutable factors that are internal to the unit, which are usually staff behaviors impacting the delivery of services, may be critical to achieve functioning QI.

QI team group dynamics. A major value of QI is to create employee ownership across the board, vertically and horizontally, to get away from a “not my job” attitude.[38] Applying numerous techniques by external experts to the functional activity does not create employee ownership. QI efforts imposed by external experts can be viewed by staff as another mandate imposed from the top down. Without buy-in and commitment from a range of staff responsible for actual implementation of QI measures, the “not my job” attitude is likely to prevail and undermine quality performance. Without buy-in and commitment, resistance to implementation is not dissuaded. Six Sigma’s emphasis on a QI expert (black belt or other belts) to direct the QI efforts may actually undermine the staff ownership that is critical for QI performance and sustainability, particularly if the expert does not have a role in the service delivery process.

Use of QI collaborative or multi-site quality council or committee. QI collaboratives or other similar mechanisms for reporting and sharing challenges and successes are exceptionally useful in (1) stimulating and encouraging a focus on results, (2) capturing and sharing lessons learned, (3) keeping focus on the main area for improvement, (4) providing visibility for benchmarks for others to emulate, (5) keeping focus on mutable factors that influence improvement (staff behaviors), (6) providing joint problem solving for common challenges outside of the control of one agency, and (7) reinforcing progress and successful performance on key metrics. Similar to the value of dashboards for publicly displaying progress and results of performance efforts, QI collaboratives provide a mechanism for visible accountability, but also provide mechanisms for supporting performance improvement. This form of QI collective can also support a major purpose of QI or what some might refer to as QI culture.

QI approaches and techniques. Some structured QI may be critical for organizing data and informing the QI team. However, overwhelming the QI team with QI techniques may be counterproductive. The use of QI techniques should support and focus on what needs to be improved, not the quantity of specific QI techniques such as Pareto graphs, control charts, and root-cause analysis that are only tools to accomplish the desired processes and outcomes. While useful to support QI efforts, too many of these tools can detract from staff ownership and even create resentment, as was observed with the Florida Immunization QI Project. They can even be perceived as bureaucratic mandates that impede rather than facilitate QI performance.

Value for IR

Although the value of QI for public health and healthcare is well recognized, its value to dissemination and implementation research, with some exceptions, is relatively unrecognized and underutilized.[13] QI can be viewed as being directly related to the primary purpose of IR, which is mainly concerned with research that enhances the adoption of evidence-based interventions. However, if the purpose of IR is only to “understand why an innovation is successfully implemented in one setting, but not in another,” where a theoretical framework provides a foundation for informing and providing insight into the implementation process, QI may not be relevant to IR.[39] QI is not intended to provide insight into the process, but rather to maximize the assets of an intervention through optimal implementation. QI has a role in IR if identifying models for implementation has utilitarian purpose of expediently and effectively getting evidence-based programs to be used by the practice community, where effectiveness is gauged by the extent that the program is implemented, rather than what made it successfully or unsuccessfully implemented.

Although there are calls for using a theoretical approach to IR and there is a substantial body of literature reviewing theoretical approaches including synthesizing a broad range of frameworks borrowed from theories such as diffusion of innovation, translational research, practice-based research, and evaluation research, little is devoted to implementation models that can be used by a broad range of practitioners.[39,40] Does this extensive and growing body of literature imply that practitioner implementation should be dependent on the theoretically informed social and behavioral science researchers, typically based in academic institutions, who are needed to study and gain insights from the implementation processes? Do the origins of QI in management science and industrial engineering preclude its use in D&I research? QI is a model for improving implementation processes that focuses on optimal success rather than insights about the process, although QI has built-in mechanisms for identifying and addressing factors that influence implementation. The fact that QI has been successfully used across an extensive range of settings should make it even more of a relevant method for D&I research.

QI as a model for implementation does require some additional research methods to document and report its process and outcomes that would be essential for it to serve as a model for research. Mixed-method evaluation research methods are particularly complementary to QI since they typically adjust the research design and methods to the project or program being evaluated. The focus of QI on process and outcome measures is also complementary to evaluation methods, and evaluation reinforces the need for QI to focus on key metrics. Combining QI with evaluation research was the basic IR design proposed for a successful Robert Wood Johnson Foundation public health systems and services research grant application involving expanded use of texting to convey STD test results to CHD clients.[35] It was also an explicit design for the community asthma projects in Georgia that were reviewed for this meta-synthesis. The merging of QI and evaluation facilitates the use of QI as a model for IR that is focused on optimal implementation but also provides the research design and methods that reflect essential observational characteristics of research.[32] In addition to assessing the success of QI in implementing an evidence-based program, robust evaluation research involving mixed-method designs and methods can also assess a myriad of other factors that may emerge from IR, including gathering insights about barriers and successes.

Adding evaluation methods to QI adds the need for additional resources, in particular additional data collection related to the use of QI procedures. Although QI tends to involve processes such as root-cause analysis to assess the causes and solutions to problems that may be impeding effective implementation, evaluation research places an even greater emphasis on the need to have consistent data collection and management related to challenges and resolution of problems such as taking notes during QI team or consortium meetings to document the use of QI processes.

Conclusion

Whereas QI methods and techniques are plentiful to the point of being overwhelming, five characteristics appeared to be critical from the case studies that were reviewed. Although specific methods may be adapted to the unique situation/context, the key characteristics of QI identified from the reviewed case studies include the following: (1) tracking, displaying, and using data for decision making, (2) focus on internal mutable factors, (3) using team/group dynamics to engage all key players involved with implementation, (4) use of some type of forum, council, or consortium for sharing and inspiring action, and (5) the use of some defined QI methods or techniques, recognizing that more may not be better. In addition to the advantages of using QI to enhance or improve healthcare and public health, QI also appears to have substantial potential as a model that can form a foundation for IR.

Limitations

It should be noted that discussion or questions related to the themes and lessons learned were frequently and consistently initiated by the researchers/evaluators, and did not necessarily emerge spontaneously or without prompts from the investigators. However, these themes were consistently reflective of QI activity. The reviewed studies were limited to a small number of case studies, consistent with meta-synthesis.[24] Consequently, the purpose was more about developing insights and not developing generalizable conclusions. The lessons learned and insights (potentially important components for effective use of QI) could form a foundation for more classic research designs to test hypotheses based on these insights or to generalize the insights to a broader range of healthcare settings or geographic regions. This might be particularly important if QI is used more extensively as a model to enhance adoption within IR.