The Methods Behind 2015 Informatics Capacity and Needs Assessment Study.

The 2015 Informatics Needs and Capacity of Local Health Departments (LHDs) survey is the most recent comprehensive source of quantitative data on LHD informatics. Conducted by the National Association of County & City Health Officials (NACCHO), this is the third nationally representative quantitative study of LHD informatics since 2009. The previous 2 comprehensive quantitative assessments were conducted by NACCHO in 2009-2010 and 2011. Given that public health informatics is rapidly evolving, the 2015 Informatics survey is a much-needed country-wide assessment of the current informatics needs and capacities of LHDs. This article outlines detailed methodology used in the 2015 Informatics survey, including instrument development, pretesting, sampling design and sample size, survey administration, and sampling weights. A 9-member advisory committee representing federal, state, and local health agency representatives guided the design and implementation of this study. The survey instrument was organized into 6 topic areas: demographics, physical infrastructure, skills and capacity available, public health workforce development needs, electronic health records, and health information exchange. The instrument was pretested with a sample of 20 LHDs and subsequently pilot-tested with 30 LHDs. The survey was administered via the Qualtrics survey software to the sample of 650 LHDs, selected using stratified random sampling. The survey was fielded for approximately 8 weeks and 324 usable responses were received, constituting a response rate of 50%. Statistical weights were developed to account for 3 factors: (a) disproportionate response rate by population size (using 7 population strata), (b) oversampling of LHDs with larger population sizes, and (c) sampling rather than a census approach.

The 2015 Informatics Needs and Capacity of Local Health Departments (LHDs) study is the third and the most recent of the series of detailed informatics studies conducted by the National Association of County & City Health Officials (NACCHO). In addition to these in-depth studies of informatics, NACCHO included a limited number of questions in its Profile of LHDs surveys that supported research investigations of the implementation of LHDs' information systems and biosurveillance.1–4 NACCHO conducted the previous 2 detailed studies of local public health informatics in 2009-2010 and 2011. The first 2 studies were closely spaced because of the increased recognition of rapidly evolving informatics, and information science, technology, and systems were among the significant drivers of modern public health.

The 2009-2010 Informatics Needs Assessment study used a Web-based survey to collect quantitative data and a series of in-person focus groups to collect more in-depth qualitative data. That study was based on a random sample of 750 LHDs, selected using stratified random sample, resulting in a response rate of 43%. The second study, the 2011 Informatics Needs Assessment, was administered to 562 LHDs using Qualtrics, a Web-based survey software, from April to May 2011. The purposes included identifying local public health readiness to exchange data from eligible health care providers and hospitals, as well as determining the type of technical assistance necessary for LHDs to meet public health informatics objectives. The sample for the 2011 Informatics study differed in that it included all 262 LHDs serving a population of 250 000+ and a random sample of 300 additional LHDs with smaller population jurisdictions. The overall response rate for the 2011 study was low (approximately 32%).

In addition to the 3 informatics studies conducted by NACCHO, another study of LHD informatics capacity was conducted by the Jiann-Ping Hsu College of Public Health (JPHCOPH), with funding from the de Beaumont Foundation. Conducted in 2014, that qualitative research study was based on 49 in-depth, hour-long interviews with LHD staff. The sample of staff members was selected using purposive variation based on geography and informatics capacities as identified in the 2013 National Profile of Local Health Departments Survey conducted by NACCHO.5 The purpose of the de Beaumont Foundation–funded study was to characterize LHDs' use of information technologies, with 2 specific aims: (a) to assess the pattern of LHDs' utilization of electronic health records, health information exchange, and other health information systems such as immunization registry, electronic disease reporting systems, and the electronic laboratory reporting; and (b) to investigate perceived philosophical, legal, cultural, and infrastructural barriers to engaging electronic systems and participation in meaningful use of electronic health records.6

Building upon these previous studies, the 2015 Informatics Needs and Capacity of LHDs study adapted questions from the validated instruments and benefited from lessons relating to sampling design, study administration approaches, and barriers to acceptable response rates. This study was a collaboration of NACCHO and the JPHCOPH at Georgia Southern University, supported by the Centers for Disease Control and Prevention (CDC). Georgia Southern University Institutional Review Board approved the study protocol in May 2015.

Survey Instrument

Advisory committee

NACCHO formed an 8-member advisory committee for the 2015 Informatics study to provide critical advice and guidance on all stages of the study, most importantly with the survey design and instrumentation. The advisory committee consisted of 8 members, including 3 LHD representatives, 3 federal partners (CDC), 1 sister organization representative from the Association of State and Territorial Health Officials, and 1 member from the Public Health Informatics Institute.

Topic prioritization and instrument development

The JPHCOPH developed the survey instrument in consultation with the advisory group. A set of topics and an initial draft of the instrument were constructed using previous instruments, expert input, and a brainstorming session with the advisory committee. Given the low response rates in the previous informatics studies by NACCHO, the advisory committee recommended that the instrument be short, which is congruent with guidelines recently proposed regarding surveys of public health practitioners. On the basis of previous studies, a list of topics was compiled and a topic prioritization survey was conducted among the advisory committee to keep the questionnaire length reasonable. The final instrument included topics with highest priority rating from the advisory committee (Table 1). The final instrument included 28 questions, resulting in a total of 271 variables.

TABLE 1 •

Topics Included in the Survey Instrument, and Topic-Specific Number of Questions and Variables

Topic	Number of Question	Number of Variables
Demographics	3	15
Physical infrastructure	5	56
Skills and capacity available	2	71
Public health workforce development needs	4	48
Electronic health records	9	32
Health information exchange	3	47
Characteristics of LHDs added to dataa	2	2
All (total)	28	271

Abbreviation: LHD, local health department.

aThe 2 variables added to the data were the size of jurisdiction population and the type of LHD governance with respect to state health agency authority (state governed, locally governed, or shared governance).

Pretesting and revisions

To improve the validity and reliability of the instrument, the research team at the JPHCOPH pretested the survey, requesting 20 LHD staff members randomly selected from all LHDs to participate in the pilot, of which 18 completed. The instrument underwent several review and feedback sessions before conducting a pretesting with 20 informatics specialists. The pretesting participants were asked to complete the survey, as well as provide their feedback about the questions. The following questions to collect participants' feedback were included at the end of the questionnaire:

Please tell us which questions do you recommend deleting from this survey and why?

Were there any questions you found confusing or difficult to understand? What recommendation(s) do you have to improve those questions?

What other recommendations, if any, do you have for us to improve this questionnaire?

Revisions and reviews were recommended by the advisory group and appropriately adjusted the instrument. The second draft of the survey instrument was pilot-tested with 30 randomly selected LHDs, asking the participants to complete the Web-based survey questionnaire and provide their feedback. The final instrument was uploaded to the Qualtrics survey software, using logical skips and displays, as well as multiple-choice, open-ended, and close-ended questions. To avoid the variation in interpretation of key terms, the research team included definitions of key terms (Table 2).

TABLE 2 •

Definition of Terms Included in the Survey Instrument

Term/Concept	Definition Provided in the Survey Instrument
Business process analysis	A systematic process by which an LHD maps out the tasks performed for specific public health operations.
Business process redesign	Rethinking the way tasks are carried out to increase the efficiency and effectiveness of public health operations.
Electronic health records	An EHR is a longitudinal digital record of a patient's care. This record may include identifiable information about individual patients, such as demographics, medical conditions, procedural history, allergies, and medications. An EHR system houses the individual EHRs.
Geographic information systems	Software used to perform spatial analysis and produce geographic visualizations such as maps.
Health information exchange	Health information exchange means the electronic transmission of health-related information between organizations according to nationally recognized standards. It does not include paper, mail, phone, fax, or standard/regular e-mail exchange of information.
Interoperability	According to HIMSS, “Interoperability describes the extent to which systems and devices can automatically exchange data, and interpret that shared data. For two systems to be interoperable, they must be able to automatically exchange data and subsequently present that data such that it can be understood by a user.”
Requirements for information system development	Requirements describe what an information system must be able to do. They can guide the selection or development of a system.
Super user	A system user who is knowledgeable enough about the system to help other users understand how to make good use of the system and perhaps has the ability to modify/customize the system.

Abbreviations: EHR, electronic health record; HIMSS, healthcare information and management systems society; LHD, local health department.

Target Population, and Sampling Design

The target population for the survey was all LHDs in the United States as defined by NACCHO in the 2013 Profile of Local Health Departments Study.5 This study used a stratified random sampling design. The stratification was performed using 7 population sizes: less than 25 000; 25 000 to 49 999; 50 000 to 99 999; 100 000 to 249 999; 250 000 to 499 999; 500 000 to 999 999; 1 000 000 or more. Potential respondents included the informatics staff to be designated by the LHD. Before sending out the survey instrument, LHD contacts in the sample were asked via e-mail to identify the most relevant informatics staff. Of the 650 LHDs in the sample, 156 provided the informatics staff contacts. The original contact was replaced with the informatics staff contact information for those 156 LHDs. The sampling distribution and response rates are presented in Table 3.

TABLE 3 •

LHDs in the Sample, Number of Respondents, and Response Rates

Population Category	Number of LHDs in the Sample	Number of Respondents	Response Rate
<25 000	209	87	42%
25 000-49 999	117	65	56%
50 000-99 999	100	43	43%
100 000-249 999	82	45	55%
250 000-499 999	56	34	61%
500 000-999 999	47	25	53%
≥1 000 000	39	25	64%
All LHDs	650	324	50%

Abbreviation: LHD, local health department.

Survey Administration and Response Rate

The survey was administered via the Qualtrics survey software to the sample of 650 LHDs. Since traditionally informatics surveys have had low response rates, rigorous follow-up was performed for this study. In the first 2 weeks, 120 completed responses were received. After 2 weeks, the first reminder was sent via an e-mail. This reminder prompted an additional 80 responses, bringing the total of completed surveys to 200. The second reminder was sent after 1 week, stating the survey would close in a week's time. This encouraged an additional 124 people to start and complete the surveys. Since the targeted response rate was still not accomplished, the survey was not closed by the originally announced date. The third reminder was sent and selective phone-based follow-up was conducted, reaching out to respondents who had started the surveys but had not completed in 48 hours since the start. In addition, we began using personalized e-mails and phone calls to follow-up with nonresponders. The survey was closed after reaching a 50% response rate. The survey fielded for approximately 8 weeks.

Data Cleaning

The survey data were downloaded and cleaned in the SPSS and STATA systems. A large number of partially completed surveys were received because some LHDs may not have found certain sections relevant (eg, those with no clinical services and for which information systems were maintained by the state health department and others). Although all LHDs were instructed that the programmed skip logic will help them skip irrelevant questions, some respondents may have determined the relevance just from the section headings. Surveys with at least section 1 of the questionnaire fully completed were included in the final data set because it was applicable to all LHDs. Survey responses with predominantly incomplete responses for the first section of the questionnaire were excluded from the final data.

The Sampling Weights

Since only a sample of all LHDs participated in the study, and the larger LHDs were oversampled (and thus overrepresented) in the sample, statistical weights were developed to account for 3 factors: (a) disproportionate response rate by population size (7 population strata, typically used in NACCHO surveys), (b) oversampling of LHDs with larger population sizes, and (c) sampling rather than the census approach. For the matrix question where the data only had “checked” versus “not checked” option, the recommended protocol was used to separate “unchecked” that meant “No” from “unchecked” that indicated missing data.