Enhancing Access to Quality Online Training to Strengthen Public Health Preparedness and Response.

OBJECTIVES: To improve access to quality online training materials developed from 2010 to 2015 by 14 Preparedness and Emergency Response Learning Centers (PERLCs) by creating quality standards and enhancing searchability through a new Web-based public health training catalog.
METHODS: The PERLC-developed training materials (n = 530) were evaluated for their capability to support development of preparedness competencies as established by 2 evidence-based competency frameworks. Inclusion/exclusion criteria and evaluation guidelines regarding training quality (design, technology, and instructional components) were systematically applied to PERLC products to create a training catalog. Twenty emergency preparedness professionals pilot tested content and provided feedback to improve catalog design and function.
RESULTS: Seventy-eight percent of PERLC resources (n = 413) met our quality standards for inclusion in the catalog's searchable database: 358 self-paced courses, 55 informational briefs, and other materials. Twenty-one training bundles were curated. DISCUSSION: We established quality guidelines, identified strengths and weaknesses in PERLC resources, and improved accessibility to trainings. Guidelines established by this work can be generalized to trainings outside the preparedness domain. Enhancing access to quality training resources can serve as a valuable tool for increasing emergency preparedness competence.

Our nation's state, local, and tribal public health agencies require ongoing training to ensure that their workforce members can maintain, improve, and adapt preparedness skills and knowledge to recurring and new types of emergencies, as well as to meet increasing expectations for preparedness capabilities.1 Evidence suggests that emergency preparedness activities of the past decade have resulted in stronger systems and better workforce skills for public health emergency preparedness and response. However, national and regional assessments report that gaps and weaknesses remain, in part due to competing demands, a shrinking workforce, and continuous budget cuts.2,3 Achieving and sustaining a strong national public health infrastructure would be greatly enhanced by the availability of rigorous and meaningful guidance that maximizes the value of limited training opportunities.

The purpose of this project was to improve public health emergency preparedness by increasing the accessibility to quality online public health training materials (trainings). The practice community can benefit from efforts to systematically compile, summarize, and make available learning products developed by the Preparedness and Response Learning Centers (PERLCs)4 and other sources. Our objective was to evaluate and organize these learning products to present the highest-quality trainings for national dissemination to state, local, and tribal public health agencies through an online catalog with features to help support enhanced decision making regarding training options. This project also sought to identify emerging areas of interest and importance in emergency preparedness competence and gaps in knowledge. To achieve these goals, within an academic-public health practice collaboration, we (a) determined which learning products fit the criteria for catalog inclusion, (b) structured a Web-based catalog to facilitate access to online trainings, and (c) organized learning products that aim to increase public health capacity to respond to emergencies. The long-term objective of this work is to support improved dissemination of PERLC trainings to strengthen the public health workforce preparedness competence.

The Public Health Preparedness Workforce Landscape

The field of public health is experiencing changes that make it simultaneously more important, but often more challenging, to have a skilled workforce for responding to public health emergencies. Both the Association of State and Territorial Health Officials and the National Association of County & City Health Officials report significant reductions in the number of public health workers in state and local governmental public health agencies.5,6 This often means that current workers have less time for training, and there are fewer staff overall to respond to emergencies. Increasing collaboration between public health and the health care delivery sector has been occurring in many parts of the country as a result of the Patient Protection and Affordable Care Act,7 encouraging care coordination and connectivity between historically disparate stakeholders to improve community-oriented population health.8 In many communities, therefore, public and community health workers are actively involved with the implementation of the Affordable Care Act.9 While these activities aim to benefit the community overall, competing demands take staff away from core public health activities related to preparedness.

As the public health workforce landscape changes, so do its training needs. Newly hired personnel require training to orient themselves to the communities they will serve. The workforce is aging and the loss of institutional knowledge and talent when older staff retire is predicted to impact public health agencies' ability to respond to both emergency and nonemergency events.10 With fewer staff, responsibilities for many positions have increased, adversely impacting retention.10 An additional pressure is the need to meet expectations associated with Domain 8 of the Public Health Accreditation Board's standards for accreditation, which require agencies to maintain a competent public health workforce.11

Supporting Preparedness Workforce Competencies and Training

Developing, applying, and refining workforce competencies and establishing standards for measuring performance, capabilities, and gaps that might be met by training and other activities have been a significant academic-public health practice agenda for strengthening the workforce. In 2010, with support from the Centers for Disease Control and Prevention (CDC), the Association of Schools & Programs of Public Health led the development of the public health workforce Preparedness and Response Core Competency Model (Core Competencies).12,13 The Core Competencies model maps work performance to competencies as appropriate to preparedness role and level of training and provides a national standard of behaviorally based, observable skills for the workforce to prepare for and respond to emergency scenarios.12 Also developed by the CDC in 2011, the Public Health Emergency Preparedness (PHEP) Capabilities12 provides a set of 15 capabilities to help state and local health departments assess and prioritize meeting strategic preparedness objectives, such as Community Recovery and Information Sharing. Alignment of competencies and capabilities with training development is intended to ensure public health preparedness and workforce response readiness. These and other initiatives were developed in response to expectations laid out in the federal Pandemic and All-Hazards Preparedness Act legislation14 that recognized interconnected national public health and medical preparedness goals, thus increasing alignment of funding sources and activities.

In 2008, the CDC funded 9 Preparedness and Emergency Response Research Centers to conduct public health systems research on emergency preparedness and response. Preparedness and Emergency Response Research Centers generated more than 130 peer-reviewed publications and nearly 80 practice tools for use in public health preparedness.15 In 2010, the CDC funded 14 PERLCs to provide emergency preparedness and response training and consultation services to state, local, and tribal public health authorities. Preparedness and Response Learning Centers were housed at academic institutions across the country (see Table 1, next section).

TABLE 1

PERLC Learning Products

PERLC Location	All Unique Learning Products Developed by PERLCs	Online Products With Active Links Included in Review	Online Products With In-active Links Not Included In Review	In-person/Other Learning Products Not Eligible for Review
	n	n	n	n
University of Albany-New York-New Jersey PERLC (NY-NJ PERLC)	71	42	10	19
University of Alabama—South Central PERLC (SCPERLC)	46	39	4	3
University of Arizona—Mountain West PERLC (MWPERLC)	40	14	10	16
University of South Florida (USF PERLC)	21	9	1	11
University of Illinois (IPERLC)	88	6	0	82
The University of Iowa—Upper Midwest PERLC (UMPERLC)	49	41	6	2
Johns Hopkins University (JH-PERLC)	61	31	10	20
Harvard University (H-PERLC)	52	27	17	8
University of Minnesota—Simulations, Exercises and Effective Education PERLC (U-SEEE PERLC)	108	42	6	60
Columbia University—Columbia Regional Learning Center (CRLC PERLC)	76	48	28	0
University of North Carolina (UNC PERLC)	244	140	80	24
The University of Oklahoma—Southwest PERLC (SWPERLC)	28	0	1	27
Texas A&M University—Training and Education Collaborative PERLC (TECS-PERLC)	52	23	29	0
University of Washington—Northwest PERLC (NWPERLC)	127	68	0	59
Total	1063	530	202	331

Abbreviation: PERLCs, Preparedness and Emergency Response Learning Centers.

The PERLC program focused on the preparedness and response training and education needs of the public health workforce. Preparedness and Response Learning Centers were tasked with developing training products that incorporated adult professional learning needs to ensure learner engagement and retention of knowledge. In addition, adult learning theory supports the creation of a learning environment in which educational objectives align with the development of specific competencies. The PERLC grantees were expected to utilize the Core Competency Model in developing trainings. Between 2010 and 2015, the PERLCs developed a total of 1063 learning products delivered in a variety of formats, including online or distance-based, face-to-face, and informational briefs, exercises and drills, and tool kits.16

While all the PERLC trainings focused on translating knowledge and information into learning tools to support workforce preparedness and response practice, the overwhelming number of trainings generated presents its own obstacle to public health agencies. Our objective was to evaluate and organize these learning products to present the highest-quality trainings for selection through an online catalog and support enhanced decision making regarding training options—the PERLC Training Catalog.

Methods

This work was accomplished over 5 phases from January 2016 to June 2017. These phases included (1) creating an inventory of candidate products for the training catalog; (2) developing an evaluation rubric to assess critical criteria; (3) developing the PERLC Training Catalog technical infrastructure and features; (4) assigning metadata to facilitate search and “bundling” trainings; and (5) usability testing.

Inventory

An inventory was compiled of all available online PERLC learning products developed between 2010 and 2015 for inclusion in the PERLC Training Catalog. Only online products were included in the inventory. Because of the lengthy time frame during which PERLC trainings were developed, an initial consideration for inclusion was currency of the information for any training. Each learning product was accessed to verify its URL. Broken or redirected links were collated and the respective PERLC was contacted for updates. During this process, PERLCs were also asked to provide information about any developed additional online trainings that could be added to the inventory for consideration in the PERLC Training Catalog. To determine ongoing availability, the team also conducted brief informational interviews via e-mail and phone with leaders from the 14 PERLCs to confirm their plans for continued availability of products and to obtain input on catalog development.

Based on input from the 14 PERLCs, we conducted a broad environmental scan of learning products and sought to identify competency-based (individual level) training and other learning products associated with PHEP Capabilities at the organization level. Extensive descriptive information was collected for each PERLC learning product including format, learning level, length, description, learning objectives, course elements, case study/scenarios, interactivity, course materials, PHEP Capabilities addressed, and core competency (individual level) area. We also assigned topic areas on the basis of core competencies to assist catalog users who were unfamiliar with PHEP Capabilities or preferred more detailed product categorization. Although PERLC course developers may have assigned multiple PHEP Capabilities to a single product, we sought to organize each course into a single primary topic area, as well as align those topic areas with the Public Health Foundation's TRAIN learning management system (LMS), which provided training links for public health. Our purpose, therefore, was to preserve learning products developed by the PERLCs and other preparedness-related training and informational resources (as with TRAIN) and to promote their availability by facilitating access through an easy-to-use interface.

Developing the evaluation rubric

Under the supervision of the Northwest Center for Public Health Practice (NWCPHP) associate director and lead investigator, a team of 3 training and curriculum design specialists and 3 graduate research assistants developed a rubric that included a set of quality criteria for design, technology, and instructional content using an interactive process of reviewing and testing. Resulting quality criteria were based on the CDC E-Learning Essentials17, the de Beaumont Foundation Public Health Workforce Continuing and Professional Education Quality E-Learning Standards,18 and existing e-learning standards and guidelines (see the Figure).

Sample Quality Criteria

The Supplemental Digital Content, available at http://links.lww.com/JPHMP/A465, illustrates our quality rubric criteria with the scoring schema. The following point values were assigned to the rating criteria: not effective (0); acceptable (1); and exemplary (2). Reliability and validity of the quality criteria rating were established using data triangulation against our models.19 The lead investigator supervised 3 graduate research assistants through the coding of 5 courses using the rating criteria; initial results revealed that agreement was achieved at a rate of 82%. The larger research team next discussed disparate results until consensus was reached and a final coding plan was established.19–21 Once scored, the coders sorted courses into 3 quality categories: low, average, and high. Products in the average and high-quality categories (n = 413) were included in the catalog while courses of high quality (n = 113) were also prioritized for use in course bundles (see Supplemental Digital Content available at http://links.lww.com/JPHMP/A465).

PERLC Training Catalog technical infrastructure and features

The PERLC Training Catalog was developed using Drupal content management software. The catalog contained 4 main elements: (1) a set of recommended training bundles, (2) a searchable training database, (3) a list of resources and emergency preparedness and response-related training sites, and (4) an “About” page. All PERLC products categorized as “high” or “average” quality, including tool kits and tabletop exercises, were included in the catalog. Users could perform open test searching as well as filter by product format, topic area, PHEP Capability, and PERLC developer. Each learning product included a detailed page with information such as format, level, university, learning objectives, and PHEP Capabilities, as well as a button linking the user to the course (when possible) on the Public Health Foundation's TRAIN Learning Network's LMS or through the PERLC developer's site. To direct users to selected PERLC online courses addressing specific PHEP Capabilities, as well as 6 additional critical skill areas, thematic training “bundles” were developed that served to focus on a series of courses in a given topic area. The bundles, established with support from our research team's physician lead, included online courses of high quality with respect to interactivity, real-world applications, and multimedia components. In some cases, however, courses categorized as “average quality” were included to fill content gaps.

Specifically, our physician lead guided the research team working from experience with formal examinations of training needs and interests and building upon our experience in asking follow-up questions about training the NWCPHP PERLC had provided. We considered as a priority those areas that were key to accreditation of state and local health departments. From those, we created an initial list of training topics for which there had been an interest in different types or levels of training.

We cross-walked this list with multiple trainings in the database looking, in particular, at topic diversity relative to different (a) intended audiences, (b) competency levels (such as “overview” or “introduction” vs “advanced” or “skills development”), and/or (c) training modalities. Then, we applied a general assessment of relative quality using our criteria for effective adult education and, when appropriate, effective online education. In the end, bundles were curated for highest quality and diversity by intended audience, level of expertise, or training modality associated with key priorities.

Usability testing

Twenty external subject matter experts (SMEs) participated in usability testing. Subject matter experts represented a variety of emergency preparedness roles, as regional emergency preparedness coordinators, partners, and coordinators from Health Resources and Services Administration–funded Regional Public Health Training Centers across the country. Subject matter experts reviewed the bundles and provided feedback to refine our approach and improve the catalog's usability. Four tested the system in person and 16 via distance through a link to a usability guide and survey from which to provide feedback. Testers were asked to browse the site and complete several specific tasks using different components of the catalog and provide qualitative feedback through 6 semistructured interview questions. Qualitative responses were organized using a thematic framework, and this input was used to improve the catalog's functionality and design.

Human participant compliance statement

According to the Human Subjects Division, University of Washington, this study did not meet the federal regulatory definition of research (institutional review board, category 2) and that human subjects' activities associated with this project were qualified as exempt.

Results

Quality scoring was used to eliminate courses from inclusion in the catalog. Table 2 describes the outcomes of applying the evaluation rubric to the eligible corpus of online PERLC products (n = 530). All products were then evaluated against the rubric and categorized as high, average, or low quality. Low-quality courses were eliminated from inclusion in the PERLC Training Catalog. The 103 archived webinars (19%) did not meet the majority of our quality criteria, such as interactivity, and so were also eliminated from consideration in the Catalog. In addition, 14 courses scored as “low quality” were further reviewed and found to be “not effective” in most criteria and/or had major access or functionality issues and so were also eliminated (3%). This resulted in 413 of evaluated PERLC resources (78%) meeting our minimum quality standards for inclusion in the PERLC Training Catalog (perlc.nwcphp.org), Of which 358 were self-paced online courses (87%) and 55 were other training materials (13%).

TABLE 2

Characteristics of Training Products Included in the Searchable Database (n = 413)

Course qualitya, n
High-quality self-paced courses (included)	113
Average-quality self-paced courses (included)	245
Other training materials (included)	55
Low-quality self-paced courses (excluded)	14
Webinars (excluded)	103
Total	530
Format, n (%)
Online self-paced course	358 (86.7)
Informational brief	43 (10.4)
Exercises and drills	9 (2.2)
Tool kits and guides	3 (0.7)
Total	413
PHEP Capabilitiesb, n (%)
Capability 1: Community preparedness	151 (36.6)
Capability 2: Community recovery	59 (14.3)
Capability 3: Emergency operations coordination	54 (13.1)
Capability 4: Emergency public information and warning	37 (9.0)
Capability 5: Fatality management	11 (2.7)
Capability 6: Information sharing	48 (11.6)
Capability 7: Mass care	27 (6.5)
Capability 8: Medical countermeasure dispensing	23 (5.6)
Capability 9: Medical materiel management and distribution	15 (3.6)
Capability 10: Medical surge	16 (3.9)
Capability 11: Nonpharmaceutical interventions	18 (4.4)
Capability 12: Public health laboratory testing	10 (2.4)
Capability 13: Public health surveillance and epidemiologic investigation	138 (33.4)
Capability 14: Responder safety and health	77 (18.6)
Capability 15: Volunteer management	9 (2.2)
Topic areas, n (%)
Epidemiology and surveillance	94 (22.8)
Bioterrorism, Chemical, and radiation emergencies	43 (10.4)
Mental health	31 (7.5)
Assessment and evaluation	26 (6.3)
Emergency management	25 (6.1)
Infectious disease and immunization	21 (5.1)
Public health preparedness basics	21 (5.1)
Risk and crisis communication	18 (4.4)
Mass care and mass fatality	16 (3.9)
Vulnerable populations	16 (3.9)
Leadership and management	14 (3.4)
Legal and ethical	14 (3.4)
Environmental health	13 (3.1)
Medical countermeasures and points of dispensing	13 (3.1)
Responder health and safety	13 (3.1)
Laboratory	8 (1.9)
Volunteer management	7 (1.7)
Incident command system	6 (1.5)
Natural disasters	6 (1.5)
Hazmat	5 (1.2)
Animal safety	3 (0.7)

Abbreviation: PHEP, Public Health Preparedness.

aNote that having fewer products included in the catalog did not necessarily indicate poorer quality of products produced by a PERLC. Some PERLCs conducted more in-person training activities and webinars, which were not included in this catalog.

bLearning products could be assigned to more than 1 PHEP capability.

To achieve a “high-quality” rating, training materials had to score “high” (2) in each of 3 categories that the research team weighed more heavily than other quality categories: interactivity, multimedia, and real-world application. Courses that received “high-quality” scores (2) across these 3 criteria, therefore, were deemed “high quality” overall (n = 113). Remaining courses were categorized as “average quality” (n = 245). Thus, weighted scoring evolved into the differentiating factor between “high” and “average” courses. Both high- and average-quality courses were prioritized for inclusion in the catalog, with high-quality products earmarked for training bundles. Other training materials included in the catalog were informational briefs (n = 43), exercises, drills, guides, and tool kits (n = 12). Since course information may be applicable to multiple topic areas, some courses were included in more than 1 bundle (n = 38). Outside of PERLC resources, we included links to additional preparedness-related resources on the catalog's “More Resources” page, such as Preparedness and Emergency Response Research Center–developed tools, the Core Competency Model site, the Public Health Foundation PERLC evaluation repository, PERLC Web sites, and the Regional Public Health Training Center Web sites.

Input from the SME pilot testers was utilized to refine the PERLC Training Catalog. One recommendation was to clearly communicate to the users what they could expect to get from the tool, as indicated by the following:

Give a more straightforward explanation of what the bundles are, and get rid of academic language like “curated.”

Users also wanted to better understand limitations for improved efficiency and ease of search and navigation. Testers indicated a preference for well-designed tools (eg, limited scrolling and/or easily searchable) and provided suggestions to interface design to support these preferences, such as,

I would consider making the search engine more front and center, it's really just a great tool.

and to improve search function usability, such as

[On the home page] the titles on the top bar match the titles in the big boxes below, except for “Search trainings” and “Searchable database.” Making it consistent may make it even more simple and easy. “Search trainings” sounds less intimidating to the average user.

Testers also indicated that they preferred searchable and bundled formats for organizing a large number of learning products, particularly when their responsibilities included selecting and disseminating training locally.

Nearly all online testers (93%) indicated that they would use the catalog in their work, and several provided examples of how they would use it. One tester shared: “I manage the (Emergency Preparedness) EP trainings for (federally-qualified health centers), and will use this information myself, as well as providing it to our membership.” Others disclosed that they planned to use training bundles to introduce staff to a new competence before giving them more advanced in-person training, or to cross-train staff in areas outside their primary responsibilities. Online testers also rated specific components of the catalog on a 5-point Likert-type scale ranging from 1 (“poor”) to 5 (“excellent”). The following components were rated as either “excellent” or “very good”: usefulness of information (79%); quantity of information (86%); organization of information (86%); ease of navigation (79%); visual appeal (71%); and overall satisfaction (86%). One tester offered, “This catalog will be helpful learning more about PHEP, HPP, CDC and public health priorities. I can utilize the catalog to have a better understanding of the requirements and the trainings available.”

Gaps and weaknesses in PERLC learning products

During the quality review and the training bundles development phases, the team identified gaps and weaknesses, such as lack of interactivity or limited use of multimedia. In addition, the team noted that most available PERLC training products were offered at an introductory level and that some PHEP Capabilities and topic areas, such as volunteer management, fatality management, and public health laboratory testing, were covered by only a few PERLC training products.

Discussion

This article reports an instructive example of an academic-public health practice collaboration, with public health academics bringing rigor, objectivity, and systematic methods to the work and public health practitioners partnering to ensure that the outcomes have “real-world” value. Public health agencies are faced with an overwhelming number of potential trainings to consider and could benefit from guidance when making the decision to improve their Public Health Preparedness and Response (PHPR) workforce's competencies and capacities. As noted in the most recent National Association of County & City Health Officials report,6 public health agencies are more likely to provide staff with emergency preparedness training than conduct the assessments that might point to a need to improve preparedness competencies. In the absence of assessments, public health agencies are less informed about how to ensure that training will fulfill an identified need or bridge a preparedness gap in their workforce. By tagging PERLC Training Catalog learning products with PHPR competencies and other metadata regarding the training, public health agencies may be able to make more informed decisions when looking for training. In addition, our work to “bundle” high-quality trainings to create modules targeted to specific PHPR topic areas raises the value proposition of training materials. In addition to providing guidance for agencies when considering training, bundling may improve uptake since a set of trainings could be viewed as a single unit. Bundling may also provide training continuity and sustainability benefits.

Yet, despite this work, gaps and weaknesses in learning products remain. As we found, the majority of trainings are introductory. While these trainings are critical for orienting new workers to public health and may be of use as “brush up” courses, few materials build up from this basic level to achieve higher levels of competence. We sought to overcome this with bundling but remained limited by the rudimentary nature of available training products. Future PHPR training efforts that focus on developing postintroductory learning products could enhance workforce competence. In addition, to widely benefit from training, public health agencies must be proficient in selecting and adapting learning products to meet workforce needs and augment agency capabilities concurrently. Academic-public health practice partnerships would be the ideal collaborative for building a “training about training” program that addresses this requirement.

The PERLC Training Catalog provides the public health workforce with quality online training resources that aim to improve PHPR competence and increase capabilities. The catalog preserves learning products developed by the PERLCs and other preparedness-related training and informational resources by promoting their availability and facilitating access through a searchable, user-friendly interface. The quality assessment approach utilized and process through which the catalog was developed are potentially applicable to training areas outside the preparedness domain and may inform future efforts to systematically organize and improve access to online learning products housed in disparate locations. Specifically, the quality review rubric may be applied to learning products in any content area, as can the process for creating training bundles.

Public health practitioners are better able to select trainings and establish a strong value proposition for investing time and effort into such training materials when clear information about a tool's content, purpose, and benefits is provided.

Compiling learning products hosted from disparate locations in a single user-friendly online catalog may facilitate their implementation in public health agencies.

By providing multiple avenues to search through and find high-quality learning products to suit different agency needs, it is possible that these training materials will be more highly utilized.

The process by which the NWCPHP's PERLC Training Catalog (perlc.nwcphp.org) was developed is potentially applicable to other training areas; future efforts to systematically organize and improve access to online learning products housed in disparate locations can be informed by this work.

In addition, the quality review criteria and the process for creating training bundles are applicable to curricula and/or coursework in other public health content areas.

Limitations and recommendations

One primary challenge our team faced was that learning products were hosted on separate PERLC developer Web sites and across different LMSs, which require users to create a new account and login credentials for each separate training developer or LMS. We addressed this challenge by including a more thorough explanation of how to access the trainings on the “About” page of the catalog Web site and by adding language above each training link informing users that they were leaving the site. However, it is unclear that this additional language would mitigate a user's frustration in accessing materials. Improving access to high-quality training materials with a single sign-on would resolve this issue. We were also challenged by frequently broken URLs as PERLCs update their own LMS or migrate to new systems. To address this, we linked to courses through the TRAIN Learning Network whenever possible, anticipating that PERLC developers would likely update URL links in TRAIN. However, it was uncertain whether the need to click through additional Web pages would deter some users from training benefits.