Quality and integration of public health information systems: A systematic review focused on immunization and vital records systems.

OBJECTIVES: Public health professionals rely on quantitative data for the daily practice of public health as well as organizational decision making and planning. However, several factors work against effective data sharing among public health agencies in the US. This review characterizes the reported barriers and enablers of effective use of public health IS from an informatics perspective.
METHODS: A systematic review of the English language literature for 2005 to 2011 followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) format. The review focused on immunization information systems (IIS) and vital records information systems (VRIS). Systems were described according to the structural aspects of IS integration and data quality.
RESULTS: Articles describing IIS documented issues pertaining to the distribution of the system, the autonomy of the data providers, the heterogeneous nature of information sharing as well as the quality of the data. Articles describing VRIS were focused much more heavily on data quality, particularly whether or not the data were free from errors.
CONCLUSIONS: For state and local practitioners to effectively utilize data, public health IS will have to overcome the challenges posed by a large number of autonomous data providers utilizing a variety of technologies.

Introduction

Both local health departments (LHD) and state health agencies need access to quantitative information for organizational decision making, strategic and community planning, as well as for the day-to-day practice of public health.1 For example, data exchanged by public health practitioners on childhood immunizations are not only used to determine if specific individuals are up-to-date on vaccinations, but as a core public health indicator, are essential for population health surveillance and for conducting quality improvement processes. Integration of their information systems (IS) is a prerequisite to having access to and sharing of real-time data across organizations with different data management systems and different data elements.

Unfortunately, major forces have conspired to prevent easy and effective data sharing in public health from being the norm. First, public health IS very much a reflection of the public health system in the US: a federation of independent states with differing relationships with their respective LHDs, tasked with various aspects of data collection, storage, and dissemination.2,3 Second, the utilized information technologies (IT) and IS in local health departments may not stem from public health needs and requirements, but instead from a local government’s broader needs, resources and existing IT/IS decisions.4 Furthermore, last decade’s relatively high level of preparedness funds awarded to public health, while providing a much needed upgrade to the IT/IS capacities of state and local agencies, has not been sustained and pre-dated the more recent advances and focus on interoperability seen in the broader health IT arena. Lastly, public health informatics is a relatively new specialty within public health and still has uneven uptake across public health agencies.5 All in all, public health organizations historically have not implemented IS or adopted IT that support efforts at efficient and effective storage and sharing of data.4,6,7

The objective of this literature review is to characterize the barriers and enablers of effective use of public health IS that support two key public health activities: immunizations and vital statistics. We purposefully selected these immunization information systems (IIS) and vital records information systems (VRIS) because they reflect longstanding key public health activities, and both have undergone recent, widespread moves toward complete electronic record keeping.8,9 Furthermore, these systems reflect different types of data, data sources, organizational involvement, and uses within the public health system. With a particular focus on supporting individual vaccinations, a comprehensive IIS includes information from public health organizations, primary care providers and even schools. In contrast, VRIS requires cooperation predominantly from local hospitals, funeral homes, and midwives, to support legal documents for individuals. The results of the literature review provide insights into the means to improve the fragmented state of public health data in the US.

Framework

This review examines the factors related to successful or unsuccessful information sharing from the dual perspectives of IS content and overall structure. Content is critical as that is the aspect of the system that is most visible practitioners relying on the IS for their daily public health work, whereas the structure of the IS in terms of data sources and technology affects the availability and quality of data in a less obvious manner for end users. We used the AIMQ’s dimensions of data quality identify and categorize attributes of the data contained within IIS and VRIS.10,11 The fifteen data quality dimensions addressed such issues as completeness, timeliness, relevancy and understandability. The structural aspect of IS intergration is aptly captured at the system-level by Hasselbring’s dimensions of IS integration: heterogeneity, distribution, and autonomy.12 The myriad of technological factors that can plague large information systems that have diverse sources of data and multiple types of organizational users fall under the heterogeneity dimension.

Problematically, barriers to interoperability originating from a lack of data and exchange standards, difficulties in networking legacy systems, and different technology platforms are commonplace in public health. For example, LHDs differ substantially in their methods of record keeping,13 and for some public health activities no national data standards exist. The distribution dimension captures the scope of the IS in terms of the number of data sources, such as local and state government agencies and private healthcare providers. The autonomy dimension includes characteristics of the organizations exchanging information as those characteristics relate to interorganizational relationships, self-determination, and governance issues. As autonomous entities, organizations and individuals may exhibit idiosyncratic behaviors around actual system usage, willingness to engage in interorganizational data sharing, or even approaches to providing public health activities.

Methods

We undertook a systematic review to identify the factors associated with successful and unsuccessful IIS and VRIS following the suggested form of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).14

Information sources & searching

We conducted a review of the English language public health and medical research literature for research, evaluation, or case studies describing experiences with the implementation, evaluation, success, usage, or failure of immunization information systems and vital records systems in the US. Searches were limited to articles published between 2005 and 2011. Both immunization information and vital record systems were searched as keyword terms in PubMed, ISI Web of Science, and the Center for Public Health Systems & Services Research Library. In addition, we reviewed the table of contents for the same time period in relevant journals, organization’s websites and conference abstracts (a sample query and all sources are provided in the Appendix). Initial search results yielded 756 unduplicated records.

Study selection

Based on abstract information, we excluded from the initial search set all non-US based studies, reviews, editorials, commentaries, and those articles that did not describe an immunization information or vital statistics system or efforts to create such, or instances where no indication existed the study was about an IS. Two members of the research team independently reviewed each record and then arrived at the excluded set through consensus. The same team members independently read the full text of each article and determined its inclusion status. Differences were resolved by consensus after a joint reading session.

Articles were retained for inclusion in the review if it described the barriers to, or factors supporting, the design, implementation, or effective use of an IIS or VRIS used by state, local or government public health agencies. These criteria allowed for the inclusion of studies addressing technical structure, data quality, or evaluations of the IS as used in practice. This excluded special purpose surveys, IS maintained solely by private providers, systems that provided only aggregated information, or articles that described the source data as part of a large research endeavor (i.e. using mortality data to supplement a hospital research study).

Data collection

From each included article, we abstracted the type of IS, study design, goals and objectives, study setting or participants, and factors identified as contributing or hindering success of the effort. We then coded each of the identified factors according to the dimensions of IS integration or data quality.

Results

Study selection & characteristics

We screened 756 unduplicated records and of those 210 warranted full text review (Figure 1). The most common reason for excluding records after full text review was that the article did not identify any barriers to, or factors supporting, either any aspect of the IS or data quality. Our review process resulted in 23 total studies of which 11 reported on immunization information system (IIS),1,15–24 11 on vital record systems,25–35 and 1 described a public health information system that integrated vital records with other information systems.36

Figure 1

Study identification and selection process.

Sixteen articles described state-level systems, but reports also included data and systems specific to Atlanta,25 Boston,20 New York city,15,21 and Philadelphia.18,19 One study interviewed immunization program managers both in urban areas and at the state level.1 In addition, the majority of included studies (17 out of 23) were cross-sectional analysis of the information system records, often compared to data generated in clinical settings or in other information systems.16–21,23–28,30–35 The remaining articles were case studies,15,22,36 a review of documents,29 and qualitative.??1

Dimensions of IS Integration

The difficulties and challenges arising from the integration of multiple technologies into a single IS were noted by six studies (Table 1); five of which were examinations of IIS. Heterogeneity was manifest as different methods of data collection,24 differences in the structure and storage of data between data sources,36 differences in the amount and quality of information collected from other systems based on technology,18–20 and differences in the IIS’ ability to both send and receive information.15 While electronic data repositories in the form of electronic health records or billing systems tended to make data more accessible to the IIS and of higher quality,19,20 simply having an electronic data source was not a panacea. For example, Kolasa and colleagues18 reported that just because an electronic billing system was in place did not mean the IIS received all the data. Furthermore, each unique system from which an IIS obtains data requires its own interface, which requires time and money.15 Conversely, Schauer and colleagues22 noted the ability of each LHDs to exchange information in Wisconsin was a strength of the IIS.

Table 1

Characteristics and findings of research studies describing public health information system factors associated with quality and system integration, 2005–2011.

Citation	System investigated	Study design & subjects	System integration issues	Data quality issues
Immunization Information systems
Kolasa, Cherry, et al. (2005)18	Philadelphia Kids Immunizations Database/Tracking System	Cross-sectional, review of IIS records & provider records	Heterogeneity Use of an electronic billing systems did not mean all data were transferred to IIS Autonomy Reporting rates varied by provider organization type & size	Completeness 24% of provider recorded immunizations not reported in the IIS
Kolasa, Chilkatowsky, et al. (2006)19	Philadelphia Kids Immunizations Database/Tracking System	Cross-sectional, review of provider records, & IIS records	Heterogeneity Differences in submission rates based on how records submitted: electronic medical records highest & electronic billing higher than paper records Autonomy Reporting rates varied by provider organization type: hospitals had highest rates	Completeness Provider charts more complete than IIS on up to date status but varied by data entry type: ○ IIS & direct entry: κ = 1.0 ○ IIS & electronic medical record: κ = 0.72 ○ IIS & electronic billing: κ = 0.66 ○ IIS & manual billing: κ = 0.65 ○ IIS & manual logs: κ = 0.42 Provider charts more complete than IIS on up to date status but varied by provider type: ○ IIS & hospital-based: κ = 0.81 ○ IIS & pediatric practice: κ = 0.37 ○ IIS & family practice: κ = 0.63 ○ Overall : κ = 0.58
Dombkowski, Leung, et al. (2007)16	Michigan Care Improvement Registry	Cross-sectional, survey of physicians		Completeness Majority of providers found IIS information complete Ease of use Few practices had difficulty in accessing IIS Free-of-error Majority of providers found IIS information accurate Value-added Concerns about patients limited to Medicaid only
Stecher, Adelman, et al. (2008)23	Arizona State Immunization Information System	Cross-sectional, review of primary care records, parental recall, & IIS records	Autonomy Mandatory reporting to IIS for providers resulted in high percent of children included Providers can choose to electronically report or mail reports	Completeness IIS specificity for up to date: 74% IIS sensitivity for up to date: 47% II positive predictive value for up to date: 72% 91% of children included in the IIS Timeliness Mailed reports may take up to 2 week to entered & electronic reports reviewed for 1-2 days before entered
Mahon, Shea, et al. (2008)20	Boston Immunization Information System	Cross-sectional, review of pediatric clinic records	Heterogeneity Fewer discrepancies in data from practices with electronic medical records Autonomy Even with an electronic medical record, providers may not create electronic records for existing patients	Believability Records of individuals older than the IIS may not be accurate Completeness Manufactures, lot numbers, doses, & dates missing in IIS Free-of-error Manufacture & lot numbers in provider records did not match IIS
White, Anderson, et al. (2009)24	Minnesota Immunization Information Connection	Cross-sectional, review of hospital records, & interviews with hospital staff	Heterogeneity IIS data derived from birth certificates, direct data entry, & historical records Autonomy Hospitals were not providing vaccination data beyond what was required on birth certificates	Completeness IIS missing birth doses administered in hospitals
Schauer, Maerz, et al. (2009)22	Wisconsin Immunization Registry	Case study	Heterogeneity All LHDs could exchange data with the IIS Distribution Difficult to identify provider participation given complexity of provider networks and affiliations	Completeness Data missing on children vaccinated by non-IIS users Free-of-error Records of children who have moved out of state not de-activated
Groom, Kennedy, et al. (2010)1	Various	Interviews 7 state & urban area immunization program managers	Distribution Care obtained from multiple providers	Ease of use Providers have quick access to IIS information
Dombkowski, Reeves, et al. (2011)17	Michigan Care Improvement Registry	Cross-sectional, analysis of reminder / recall notifications sent by LHDs		Timeliness Outdated parental contact information
Papadouka, Metroka, et al. (2011)21	New York City Immunization Information System	Cross-sectional, review of patient records by LHD staff		Completeness Only 37% of doses recorded in IIS contained lot numbers
Arzt, Forney, et al. (2011)15	New York City Citywide Immunization Registry	Case study	Heterogeneity Requires multiple interfaces, “Each vendor requires a separate development effort and a significant investment of time and resources” Primarily unidirectional data flow & few vendors prepared to implement bi-directional data flow
Vital Records Information Systems
Smith, Veazie, et al. (2005)35	Maryland vital records	Cross-sectional, comparison of death certificates & multiple occupational injury fatality systems		Free-of-error Death certificates had a higher sensitivity than the other sources (89%), but still did not include all cases
Lydon-Rochelle, Holt, et al. (2005)32	Washington vital records	Cross-sectional, comparison of birth certificates, hospital discharge data, & medical records		Free-of-error Birth certificate maternal labor & birth events: ○ True positive fraction ranged from 34.4% to 81.2% ○ False positive fraction ranged from 0.1% to 7.4% ○ Positive predictive values ranged from 60.3% to 93.0% ○ Negative predictive values ranged from 81.2% to 99.1%
Mann, Knight, et al. (2005)33	Utah vital records	Cross-sectional, comparison of death certificates, hospital records, & emergency medical services records		Free-of-error Death certificates do not reflect all injury mortalities
Rodriguez, Mallonee, et al. (2006)34	Oklahoma vital records	Cross-sectional, comparison of death certificates & injury surveillance system data on traumatic brain injury mortality		Free-of-error Sensitivity of death certificates = 78% Positive predictive value of death certificates = 98% Higher odds of missed cases for females & older individuals Higher odds of missed cases by cause of death coding
Horon (2005)30	Maryland vital records	Cross-sectional, comparison of death certificates, fetal death records, & medical examiner records		Free-of-error 38% of maternal deaths were unreported on death certificates
Lydon-Rochelle, Cárdenas, et al. (2005)31	Washington vital records	Cross-sectional, comparison of fetal death records with medical records		Free-of-error Fetal death certificate maternal & perinatal conditions: ○ True positive rate ranged from 0.0% to 100.0% ○ False positive rate ranged from 0.0% to 10.5% ○ Positive predictive values ranged from 0.0% to 100.0% ○ Negative predictive values ranged from 33.3% to 100.0%
Caveney, Smith, et al. (2006)27	Texas vital records	Cross-sectional, comparison of death certificates, medical records, & individual interviews		Free-of-error Death certificate race/ethnicity reporting consistent with self-reports ○ Percent agreement: 97.1% ○ Sensitivity: 95.4% ○ Specificity: 99.0%
Fiscella & Meldrum (2008)28	California vital records	Cross-sectional, comparison of death certificates & hospital discharge records		Free-of-error Agreement between hospital discharge records & death certificates variable by race: ○ White: κ = 0.76 ○ Black: κ = 0.92 ○ Asian: κ = 0.88 ○ Native American: κ = 0.27 ○ Other: κ < 0.01
Brender, Suarez, et al. (2008)26	Texas vital records	Cross-sectional, comparison of birth certificates & individual interviews		Free-of-error Birth certificates have high specificity with parental self-reported occupation Sensitivity of birth certificates for maternal & parental varies by occupational classification group
Fitzgerald, Wartenberg, et al. (2009)29	50 states’ birth & fetal death records	Document review of forms		Relevancy Most states did not collect information that would be useful for environmental exposure investigations: ○ Duration of residence for mothers ○ Paternal residence ○ Parental occupation
Boulet, Shin, et al. (2011)25	Atlanta vital records	Cross-sectional, comparison of birth certificates & Metropolitan Atlanta Congenital Defect Program records		Free-of-error Low sensitivity of birth certificates for birth defects
Multiple content systems
Chapman, Ford, et al. (2011)36	Virginia Vital Events and Screening Tracking System	Case study	Heterogeneity Differences in data field types between systems Autonomy Organizations create different identifiers for the same individual

Regardless of what technology is used to share data with public health agencies, data sharing remained subject to variation introduced by individual users and organizations. Organizational and individual autonomy are illustrated at a very basic level by differences in the rate of data sharing by organizational type and size.18,19 The challenges posed by differences in organizational practice were illustrated in three studies. Specifically, organizations create different identifiers for the exact same individual,36 individuals within an organization may choose not to use an information system,20 or organizations may not collect all the data desired by public health organizations.24 As a potential solution to the problems posed by organizational and individual autonomy in data sharing, the Arizona State Immunization Information System reported a higher percentage of children were included in the IIS due to mandatory reporting for providers.23

Only two studies described how the number of data sources, or the problem of distribution, affected public health data sharing. A case study of the Wisconsin Immunization Registry noted the basic activity of identifying provider participation in the IIS was complicated by the number of networks and affiliations contributing data to the system.22 Also specific to data on immunizations, Groom and colleagues1 noted how the reality of patients seeking care from multiple providers can prevent public health from having complete information.

Data quality issues

The studies of vital record information systems (VRIS) were exclusively concerned with data quality and more specifically if the data were free-from-errors. Under the data quality assessment framework, free-from-error is roughly analogous to misclassification bias. So most studies were not concerned whether or not a vital event had occurred, but whether the data describing that event was error free. In terms of the cause of death, VRIS underreported injury mortality33–35 and VRIS also did not accurately reflect pregnancy status30 on death certificates. Natality data in the VRIS were also the subject of reviewed studies, which concluded substantial variation on the recording of labor & delivery events,32 birth defect data were not free-from-errors,25 and parental occupations were not categorized correctly.26 The reviewed studies were not conclusive about the extent race/ethnicity data in VRIS are free-from-error. For example, in a comparison of California hospital discharge records and death certificates, Fiscella and Meldrum28 noted wide variation in the data agreement by race/ethnicity, but a comparison of death certificates with self-reported race/ethnicity found high agreement in Texas.27 Lastly, Fitzgerald and colleagues’29 warrants specific attention because of all the VRIS studies, it was the only one concerned with the relevancy of the data. Instead of assessing whether data were free-from-error or complete, the authors sought what types of additional data would be useful.

The reviewed articles examining IIS tended to be more diverse in their treatment of data quality issues than those focused on VRIS. However, if the data were complete and free-from-error was still a major focus. Studies reported their respective IIS were missing vaccination doses,18–20,23,24 missing manufactures & lot numbers,20,21 or even whole individuals.22 Even when present, manufacture and lot numbers may be wrong20 or individuals may have moved out of the state.22 Other dimensions identified in the reviewed articles included: the believability of older records in IIS,20 the timeliness of the available contact information,17 and the ease of system use.1

A single article requires additional comment. Dombkowski and colleagues16 presented a more comprehensive evaluation of an IIS than the other studies by examining end user perceptions. While these were the perceptions of physicians users, the system was nonetheless a public health IIS. They reported positive perceptions of data completeness, that it was free-from-error, the system was easy to use and that it was beneficial to use in their work.

Discussion

The review of evidence on IS in public health, as revealed through immunization and vital records, confirms the general perception that significant challenges and barriers prevent public health from leveraging IS to its fullest extent.6 In general, this review revealed public health IS are struggling with issues of integrating multiple technologies, data sources and independent organizations and often not providing information of sufficient quality to public health practitioners. The IS qualities reported had bearing on public health practice, policy making and research. However, beyond simply documenting challenges, this review provides insights into the interrelated nature of system features and data quality issues and highlights the need to incorporate more end user perspectives.

While each dimension of system integration and data quality were assessed independently, the articles included in our review on IIS illustrate how one dimension can affect another. For example, autonomy may be highly correlated with system heterogeneity as was the case in Arzt and colleagues’15 report where independent organizations were making their own, and very different, technology vendor choices. Likewise, several studies noting examples where organizations limited data sharing, choose inefficient methods of exchange or did not participate fully in the system also reported issues around completeness.18,23,24 These instances of the importance organizational and individual decisions on an integrated IS, or at least a hoped for integrated IS, indicate both a priority and a role for public health. Although data quality is paramount for quality decision making, the priority in public health IS should be from the system perspective. Because public health deals with populations, if differing technologies are not rectified, if individual differences in organizational practices are not mitigated and if all components of the public health system not included, it will not be possible to compile data that is accurate, timely, complete and error-free. Without a systems focus, the dimensions of IS integration (heterogeneity, distribution and autonomy) are ignored and thus likely to present as longer term, but avoidable, problems.

To facilitate this approach, public health agencies need to take a strong coordinating role. We intentionally use the word coordination, because no public health entity will be able to eliminate organizational autonomy. Even in states with highly centralized governance structures, public health will not be able to dictate specific vendor choices or business processes. Current federal policy around electronic health records demonstrates that reality. What public health can do is coordinate around specific capabilities like exchange standards and required data elements, which is consistent with broader health IT policy, provide guidance to locally governed LHDs, or deploy uniform solutions within centrally governed states.

Previous research agendas in the area of public health IS noted the importance of attending to user needs.37,38 With a few exceptions (for example1,16) specific public health user perceptions or needs were not at the forefront of the reviewed literature. However, the examined literature revealed that public health data are valued as much for research purposes as for practice meaning the authors were primarily focused on researchers as their users of interest (for example20,27,28,30). A dichotomization between the practice and research value of public health information does not bode well for generating evidence relevant to understanding the real-time, daily needs of LHD staff, for participatory research with practitioners, nor for generating knowledge from practice. Researchers are more likely to be concerned with quality of data than on integration, compared to the concerns of practitioners. Both groups ought to be equally concerns about both aspects of IS.

This review documents the state of two public health IS examples over the past six years. However, public health IS will undoubtedly undergo a transformation over the course of this decade for two very different reasons. The first is essentially self-imposed: the launch of the public health accreditation process. Accreditation standards focus on data driven decision making and include requirements of data sharing with other public health organizations,39 which should foster interests and investment in public health IS. With the majority of state public health agencies intending to seek accreditation40 and not an inconsequential level of interest among LHDs,5 the potential for increased attentiveness to IS issues around the nation could increase substantially. The other development, which will have a multi-faceted impact, is the national effort to increase adoption of electronic health records under the Meaningful Use program.41 For LHDs providing primary care services, this program will provide unparalleled financial incentives to adopt a key, interoperable clinical information system thereby increasing the technological capacity for numerous public health organizations. For all of public health, the Meaningful Use criteria specific to public health reporting will dramatically increase existing distribution and heterogeneity challenges. Effectively all healthcare providers and organizations will soon be able, and required, to contribute data to public health. Public health will have unprecedented access to clinical data on essentially everyone who accesses the healthcare system, but it will come at the cost of a dramatic increase in number of organizations contributing to public health as well as the data originating from numerous technologies.

Conclusions

For state and local practitioners to effectively utilize data, public health IS will have to overcome the challenges posed by a large number of autonomous data providers utilizing a variety of technologies. Efforts to ensure quality information must remain attentive to the role of overarching system factors.

Table 2

Summary of Distribution of Issues by Assessment Framework (n=23 studies).

Dimensions of IS Integration	Immunization	Vital Records	Multiple content
Heterogeneity	6		1
Autonomy	5		1
Distribution	2
Data Quality Characteristics	Immunization	Vital Records
Completeness	8
Error Free	3	10
Ease of use	2
Timeliness	2
Believability	1
Value added	1
Relevancy		1