Public Health Quality Improvement Exchange: A Tool to Support Advancements in Public Health Practice.

OBJECTIVES: The Public Health Quality Improvement Exchange(PHQIX) is a free, openly available online community that supports public health practitioners in the rapidly evolving landscape of public health quality improvement (QI). This article's objective is to describe the user-centered development of PHQIX and its current content and examine how elements of a QI initiative may vary by an organization's characteristics or QI experience.
METHODS: PHQIX was developed by taking a user-centered iterative design approach, seeking early and continued input from users to gather requirements for the website. We performed an exploratory analysis of the published QI initiative descriptions, reviewing all QI projects that PHQIX users shared as of January 1, 2018.
RESULTS: PHQIX features 193 QI initiatives from a variety of health departments and public health institutes using a wide range of QI methods and tools. DISCUSSION: Submitted QI initiatives focus on many public health domains and favor the PDCA/PDSA cycle; Kaizen; and fishbone diagrams, flowcharts, process maps, and survey methods. Limitations include data coming only from users who represent health departments with sufficient time to complete the PHQIX submission template. Additionally, many initiatives were submitted in part to fulfill a grant requirement, which could skew results.
CONCLUSION: As the field of QI in public health practice evolves, resources targeted to QI practitioners should build on and advance the available resources. Findings from this study will provide insight into QI initiatives being performed and the types of projects that can be expected as organizational experience and collaboration grow.

Introduction

Quality improvement (QI) has become an increasingly important activity for health departments as they seek to efficiently improve the health of populations they serve. Public health departments are implementing QI initiatives rapidly, given these initiatives’ potential to streamline processes, reduce costs, improve health outcomes for populations, and implement a culture of quality in organizations [1-3]. Health departments are also moving toward QI to document capacity and performance standards as part of the process of accrediting their organizations [4].

The Public Health Quality Improvement Exchange (PHQIX) is a free, openly available online community that supports public health practitioners in the rapidly evolving landscape of QI in public health [5]. Supported by the Robert Wood Johnson Foundation, this effort has published 193 in-depth descriptions of real-world QI initiatives on its website since it launched in October 2012. PHQIX is unique among resources for public health practitioners because of its exclusive focus on QI in public health, QI project documentation, its community features, and its integrated search capacity that allows users to find examples of recent QI work from other health departments around the country [6].

The primary resource on the PHQIX website is the set of initiative descriptions that present detailed data in a structured format. The descriptions may also include attachments such as diagrams, QI tools and process templates, surveys, policies, and storyboards. Users can browse and search these descriptions through free-text search or by using a set of faceted terms based on a public health QI taxonomy developed specifically for the website. For example, users can search for projects in a variety of topical categories such as immunizations, laboratory services, maternal and child health, environmental health, and administrative areas. Although the QI initiative descriptions are not intended to represent best practices, they help practitioners learn from previous work performed at other health departments and allow them to adopt or adapt elements of the initiatives for their own purposes. Users frequently seek out initiatives that would be practical to implement at their own health departments and are interested in using products of these initiatives (e.g., storyboards, surveys) to increase QI capacity and accreditation readiness at their institutions [7].

The aim of this study is to describe the user-centered development of the tool and its current content and to examine how elements of a QI initiative may vary by an organization’s characteristics or QI experience. The content descriptions include an initiative’s methods (e.g., Lean/Six Sigma, Kaizen), tools (e.g., surveys, process maps), and focus activities (e.g., immunizations, data collection, administrative activities).

Methods

PHQIX was developed by taking a user-centered iterative design approach [8-11], which sought early and continued input from users to gather requirements for the website, test early prototypes, and participate in an evaluation of the site [7]. Early in the project, we formed two sets of experts to guide the design and development of the system. Those working in QI in public health became members of our User Group (UG), and national-level experts in QI became members of the Expert Panel (EP). The UG and the EP participated in focus groups; from these groups, we developed storyboards to reflect requirements and design elements for the system. The storyboards were vetted through the EP and were used to guide the development of the initial prototypes, which were tested by stakeholders at public health conferences. These stakeholders were selected because they represented the intended users of the system, and they were identified through the EP.

Feedback about the usability of the prototypes was sought using the think-aloud process while stakeholders interacted with the prototype by working through real-life scenarios. This feedback was then incorporated into the next prototype version, which was then tested with users again. A final iteration of the prototype was produced and launched in October 2012 at the American Public Health Association Annual Meeting and Expo. After the rollout of the first operational system in 2012, limited development occurred over the next 2 years, and in 2014, the system went into operations and maintenance mode. Community supporting features that were added during those 2 years included a map of submitters’ locations; QI Spotlight articles, featuring aspects of QI work in public health; video highlights of exemplary projects; Ask An Expert Q&A; the Community Forum threaded discussion list and associated incentives for participation; the monthly newsletter detailing information about the community and the project; and the weekly digest, providing quick snapshots of information about QI happenings.

Our exploration of the published QI initiative descriptions consisted of a review of all 193 public health QI projects and the organizational characteristics that registered PHQIX users shared, as of January 1, 2018, approximately 5 years after the website’s initial launch. The users who submitted the QI initiative data were employees of state, local, and tribal health departments who had participated in a public health QI activity. Submissions are subject to a review process by the EP, composed of QI subject matter experts. Upon receiving new submissions, the site’s submission coordinator confirms that all appropriate data fields are properly completed, then distributes the initiative to an expert panel member for review. The panel member then assesses the relevance of the submission as a true QI initiative and determines whether sufficient documentation is provided. Finally, the panel member works directly with the initiative’s submitter to make any necessary clarifications or additions. All accepted submissions are then published on the website and have been included in this study population. The unit of analysis for this study is the QI initiative or project.

We conducted an exploratory analysis of the data fields across the QI initiatives, examining what implications an organization’s characteristics have for the type of initiative it will conduct. We sought to determine whether health departments of a particular size, type, or experience with QI would affect the likelihood of them using specific tools or methods or selecting a specific area of services for improvement. The exploration of the initiatives included information about the health department and the population it serves, tools and methods used, project duration, types of partner organizations with which the health department collaborated, and the health department’s level of QI activity. For any QI initiative, it is possible to have used multiple methods, tools, or focus areas. We used the chi-squared test to confirm all statistically significant associations among variables. This paper reports descriptive statistics of these self-assigned attributes and the results of multivariate analyses among QI methods, tools, and health department QI activity level.

Results

The PHQIX website features 193 QI initiatives from 38 states (Figure 1), which included 159 submissions from local health departments, 28 from state health departments, 2 from tribal health departments, and 4 from public health institutes. Washington State and North Carolina have produced the most published submissions, with 16 initiatives each. Oregon (13), Wisconsin (13), Illinois (12), and Michigan (11) all also published more than 10 submissions. The Midwest and Pacific Northwest regions have had more PHQIX submission activity than other regions, whereas 12 states have not published QI initiatives on PHQIX. The self-reported organizational QI activity level at the submitting organization varies in an ordinal range including formal QI in specific areas (39.9%), informal QI (21.2%), formal agency-wide QI (20.7%), QI culture (11.9%), and QI community (3.6%) [12]. The organizations serve populations ranging from fewer than 24,499 people (5.2%) to more than 1 million (17.1%). The most common population groups are 100,000 to 249,999 (23.3%), more than 1 million (17.1%), and 250,000 to 499,999 (16.2%). The most common submitting organization type is county health department (45.6%), followed by state health department (14.5%), city-county health department (7.8%), and multi-county health department (6.7%).

Figure 1

Published PHQIX QI initiatives, by state: 2012–2017

The data show that submissions come from a wide variety of public health departments, with smaller health departments submitting fewer initiatives (26.9%), and health departments serving more than 100,000 people submitting more initiatives (73.1%). Health departments with a QI activity level of at least “Formal QI in Specific Areas” submitted the majority (76.2%) of published initiatives, whereas health departments with “Informal QI” submitted only 23.8% of the total number.

Table 1 shows that although the Plan, Do, Check/Study, Act (PDCA/PDSA) cycle is the most commonly used QI method or approach (88.1%), health departments also use Kaizen (19.2%), rapid-cycle improvement (17.1%), Lean/Six Sigma (13.5%), and Model for Improvement (13.0%). The most frequently used QI tools reported in Table 2 were brainstorming (67.9%), fishbone diagrams (54.4%), process maps (47.2%), flowcharts (45.6%), surveys (42.5%), root cause analyses (39.4%), and the 5-Whys (32.6%). The most commonly reported partner organizations of the leading organization conducting the QI initiatives were local health departments (20.7%), state health departments (16.1%), community-based organizations (9.9%), and universities (6.2%).

Table 1

QI methods/approaches for published QI initiatives (N=193)*

QI Method/Approach	Number of Initiatives	Percentage
Plan, Do, Check/Study, Act cycle	170	88.1%
Kaizen	37	19.2%
Rapid-cycle improvement	33	17.1%
Lean/Six Sigma	26	13.5%
Model for Improvement	25	13.0%
Nominal group technique	5	2.6%
Business process analysis	4	2.1%
Adaptive promising practice	1	0.5%
Standardize, Do, Check, Act cycle	1	0.5%
Total quality management	1	0.5%

*The methods/approaches listed in this table are not mutually exclusive; therefore, the sum of percentages exceeds 100.0%.

Table 2

QI tools for published QI initiatives (N=193)*

QI Tool	Number of Initiatives	Percentage
Brainstorming	131	67.9%
Fishbone diagram	105	54.4%
Process map	91	47.2%
Flowchart	88	45.6%
Survey	82	42.5%
Root cause analysis	76	39.4%
5-Whys	63	32.6%
Cause-and-effect diagram	56	29.0%
Prioritization matrix	49	25.4%
Affinity diagram	45	23.3%
Pareto chart	22	11.4%
Run chart	21	10.9%
Multi-voting technique	17	8.8%
Check sheet	14	7.3%
Force-field analysis	13	6.7%
Histogram	8	4.1%
Strengths-Weaknesses-Opportunities-Threats analysis	8	4.1%
Control chart	7	3.6%
Interrelationship digraph	6	3.1%
Radar chart	6	3.1%
Control and influence plot	5	3.6%
Tree diagram	4	2.1%
Process decision program chart	3	1.6%
Specific, Measurable, Achievable, Realistic, Timely (SMART) chart	3	1.6%

*The tools listed in this table are not mutually exclusive; therefore, the sum of percentages exceeds 100.0%.

QI methods and tools do not vary widely by population served, type of health department, or level of QI activity. PDCA/PDSA methods are used widely across all organization characteristics, but health departments serving larger populations use the Kaizen method more (30.9% for populations greater than 500,000) compared with those serving smaller populations (12.7% for populations fewer than 500,000) (p=.005). Health departments for larger populations are also slightly more likely to use Lean/Six Sigma (17.6% for populations greater than 250,000) than those for smaller populations (8.3% for populations fewer than 250,000). The same is true for rapid-cycle improvement (21.2% for populations greater than 250,000 and 14.6% for populations fewer than 250,000), although these observations did not achieve statistical significance.

Health departments submitting to PHQIX also indicate the types of organizations they partnered with during their QI initiative. Although the most common partner organization types are not significantly affected by submitting organization type or organizational QI activity level, state health departments are more likely to have a partner organization for their QI initiative. State health departments partner with local health departments (p=.02), other state health departments (p=.002), and community-based organizations (p=.001) in 36%, 36%, and 39% of initiatives, respectively. Meanwhile, local health departments partner with the same organization types at rates of 16%, 12%, and 5%, respectively.

Table 3 shows the most common focus areas for published initiatives: policies/internal procedures and processes, QI and accreditation readiness, organizational effectiveness, and customer service/satisfaction, all in the Administration category. In addition, the data indicate that organizations with a higher organizational QI activity level reported with higher frequency that they perform initiatives focused on policies/internal procedures and processes and QI and accreditation readiness.

Table 3

Most common focus activities for QI initiatives submitted by health departments (N=193)

Focus Activity	Number of Initiatives	Percentage
Policies/internal procedures and processes	40	20.7%
Organizational effectiveness	37	19.2%
QI and accreditation readiness	34	17.6%
Customer service/satisfaction	32	16.6%
Access to care	25	13.0%
Data collection and management/information technology	22	11.4%
Environmental health	22	11.4%
Communications	20	10.4%
Women, Infants, and Children programs	16	8.3%
Workforce development	16	8.3%
Communicable/infectious diseases	15	7.8%
Prenatal care	14	7.3%
Capacity development	14	7.3%
Performance management	13	6.7%
Childhood immunizations: Administration of vaccine to population	12	6.2%
Maternal and child health (data collection, epidemiology, and surveillance)	12	6.2%
Reportable diseases	10	5.2%
Collaboration/resource sharing	10	5.2%
Maternal and child health home visits	9	4.7%
Food safety education	9	4.7%
Financial management	8	4.1%
Tobacco	8	4.1%

As shown in Figure 2, the number of QI initiatives an organization reports performing per year appears to be associated with that organization’s self-reported organizational QI activity level. Figure 2 illustrates that of the organizations reporting informal QI, 83% perform only one to three initiatives per year, whereas only 29% of those reporting a QI community perform the same number of initiatives. Additionally, 0% of organizations reporting informal QI perform 7 to 10 initiatives per year, whereas 43% of those reporting a QI culture do so. Similarly, the percentage of organizations that perform 11 to 20 initiatives annually rises from 0% to 13% for the same groups. The informal QI organization set is also the only one with health departments that perform no QI initiatives per year (10%). Organizations with less formal QI undertook fewer QI initiatives in a given year. The most common duration of QI initiatives submitted to PHQIX, measured from start to finish, is 6–12 months (52.8%), followed by less than 6 months (21.2%) and 12–18 months (7.8%).

Figure 2

Annual number of QI initiatives, by organizational QI level: 2012–2017

Discussion

The PHQIX website and community were designed and developed with engagement of a broad group of public health and QI stakeholders. This ensured that the website would meet the needs of the intended user audience, which is largely composed of local and state public health departments and institutes. In examining those that submitted QI projects to PHQIX, we found that local health departments are much more likely to submit initiatives than state health departments. This is probably because of the significantly higher proportionate number of existing local health departments and because of increased interaction with PHQIX from local health departments that received QI grants from the Robert Wood Johnson Foundation. A higher percentage of submissions were received from staff at health departments working to formalize QI at their health department, which may be the result of increased efforts to perform and document QI work as they strive for accreditation. Conversely, health departments with informal QI perform fewer QI initiatives and may be less inclined to share their work, feeling that it may not meet the standards of QI performed by health departments with more formalized QI. State health departments showed a higher likelihood of collaboration with a partner, which may be caused by many factors, including assisting local health departments, a larger project scope, or more initiative stakeholders.

The submitted QI initiatives focus on many different public health domains and favor the PDCA/PDSA cycle; Kaizen; brainstorming; and using fishbone diagrams, flowcharts, process maps, and survey methods. Supporting similar national findings of local and state health departments, the PDCA/PDSA cycle remains the predominant choice for QI method [13,14]. (insert citations of ASTHO and NACCHO Profile data) More recent submissions indicate growing use of the Kaizen method (particularly for organizations that serve larger populations), and future research should monitor this trend. Health departments serving larger populations may be more likely to use such methods as Kaizen and Lean/Six Sigma, perhaps because they have funds budgeted for trainings or seek grants featuring such methods. Although no significant relationships are evident among health departments’ type, capacity, or the methods and tools employed, an organization’s reported QI activity level may have an association with the number of QI initiatives it performs annually and the duration and focus of those initiatives. Health departments seeking to formalize QI will continue to increase the number of initiatives performed annually. .

Conclusion

As the field of QI in public health practice evolves, resources targeted to QI practitioners should build on and advance the available resources. The increasing number of health departments across the country seeking accreditation will continue to fuel interest in QI information and trainings. As more health departments are likely to increase the formalization of QI at their organizations, future research should continue to monitor the trends of initiatives from organizations with a growing QI culture. Understanding the trajectory of the field of QI in public health is important for practitioners and researchers alike. Findings from this study will provide insight into QI initiatives being performed and the types of projects that can be expected as organizational experience and collaboration grow. As previous studies used QI initiative data to establish a framework to define and assess the impact of QI [15] and to determine which characteristics of QI projects affect whether a given project will achieve its stated goals [16], collecting measures of efficiency or effectiveness could help to expand the usefulness of the database.

PHQIX is now being transitioned to the Public Health Accreditation Board [17], which will become the host of the resource.

Limitations

This study had several limitations: the first is that the data come only from users who represent health departments that could dedicate the time to complete the PHQIX submission template, along with making any necessary revisions. As a result, the PHQIX database represents a snapshot of QI activities in public health, and it is unknown whether that snapshot is representative of the universe of public health QI projects. Many of the initiatives described were submitted in part to fulfill a grant requirement, which could skew the data if only specific types of health departments were eligible for this grant funding or if such grants focused on implementing a specific QI method or tool. Additionally, although a panel of QI experts review the submissions, the accuracy of some of the collected data fields is reliant on the submitter’s understanding of the various options (e.g., health department’s organizational QI activity level) and are therefore subject to errors of self-report. Finally, although the database of QI initiatives featured on PHQIX is substantial, it may not be large enough to infer statistical significance in all observed trends and associations where true differences exist.