Evaluation of Data Exchange Process for Interoperability and Impact on Electronic Laboratory Reporting Quality to a State Public Health Agency.

BACKGROUND: Past and present national initiatives advocate for electronic exchange of health data and emphasize interoperability. The critical role of public health in the context of disease surveillance was recognized with recommendations for electronic laboratory reporting (ELR). Many public health agencies have seen a trend towards centralization of information technology services which adds another layer of complexity to interoperability efforts.
OBJECTIVES: The study objective was to understand the process of data exchange and its impact on the quality of data being transmitted in the context of electronic laboratory reporting to public health. This was conducted in context of Minnesota Electronic Disease Surveillance System (MEDSS), the public health information system for supporting infectious disease surveillance in Minnesota. Data Quality (DQ) dimensions by Strong et al., was chosen as the guiding framework for evaluation.
METHODS: The process of assessing data exchange for electronic lab reporting and its impact was a mixed methods approach with qualitative data obtained through expert discussions and quantitative data obtained from queries of the MEDSS system. Interviews were conducted in an open-ended format from November 2017 through February 2018. Based on these discussions, two high level categories of data exchange process which could impact data quality were identified: onboarding for electronic lab reporting and internal data exchange routing. This in turn comprised of ten critical steps and its impact on quality of data was identified through expert input. This was followed by analysis of data in MEDSS by various criteria identified by the informatics team.
RESULTS: All DQ metrics (Intrinsic DQ, Contextual DQ, Representational DQ, and Accessibility DQ) were impacted in the data exchange process with varying influence on DQ dimensions. Some errors such as improper mapping in electronic health records (EHRs) and laboratory information systems had a cascading effect and can pass through technical filters and go undetected till use of data by epidemiologists. Some DQ dimensions such as accuracy, relevancy, value-added data and interpretability are more dependent on users at either end of the data exchange spectrum, the relevant clinical groups and the public health program professionals. The study revealed that data quality is dynamic and on-going oversight is a combined effort by MEDSS Informatics team and review by technical and public health program professionals.
CONCLUSION: With increasing electronic reporting to public health, there is a need to understand the current processes for electronic exchange and their impact on quality of data. This study focused on electronic laboratory reporting to public health and analyzed both onboarding and internal data exchange processes. Insights gathered from this research can be applied to other public health reporting currently (e.g. immunizations) and will be valuable in planning for electronic case reporting in near future.

Introduction

Past [1] and present [2] national initiatives that promote electronic health records (EHRs), also advocate for the electronic exchange of data across various healthcare sectors using nationally recommended standards [3]. The critical role of public health, in the context of disease surveillance is recognized by these regulations, with recommendations for electronic laboratory reporting (ELR). ELR refers to the electronic transmission of labs related to reportable conditions to public health [4]. The emphasis on interoperability in recent legislations [5] and roadmaps [6] is facilitating the focus on electronic movement of data across healthcare settings. Many public health agencies have seen a trend towards centralization of information technology services which adds another layer of complexity to interoperability efforts. Given this landscape, it is essential to understand the process of data exchange and its impact on quality of data being transmitted, as this is a crucial step in interoperability. In addition, this holds broad implications for future priority transactions such as electronic case reporting to public health.

Initial research around ELR focused on comparison of paper-based reports to electronic transmissions and found predominantly positive impact of ELR [7,8] on specifically two metrics of data quality: timeliness and completeness. Subsequent studies have assessed the role of intermediaries such as Health Information Exchanges (HIE) [9-11] to facilitate ELR and reported better completeness of data with HIE support. Presently, studies have begun to focus on provider reporting of notifiable diseases [12,13], as moving to electronic case notification [14-16] along with ELR will be great progress to support overall public health disease surveillance. Challenges in adoption and use of recommended codes [17-19] and need for an informatics savvy workforce [20] were identified as some of the issues in the move towards ELR [21].

A recurring theme across these studies was assessing the quality of data, including exploring new venues to measure [22-24] and improve [25] it. Timeliness and completeness were the two dimensions of data quality (DQ) which were often evaluated. Metrics from DQ frameworks published in literature can be used as guidance in identifying additional parameters for assessment. Data quality assessment framework by Kahn et al. [26], identifies three DQ categories: conformance, completeness and plausibility, along with verification and validation as two DQ assessment contexts. DQ framework by Strong et al., proposes a broad conceptualization of the quality of data from perspective of data consumers. It defines high quality data as one that is fit for use and emphasizes context around data production and usage.

Strong’s framework proposes four DQ categories (Intrinsic DQ, Contextual DQ, Representational DQ, Accessibility DQ) comprising of fifteen DQ dimensions [27,28]. These include Intrinsic DQ (Accuracy, Objectivity, Believability, Reputation); Contextual DQ (Relevancy, Value-Added, Timeliness, Completeness, Amount of data); Representational DQ (Interpretability, Ease of understanding, Concise representation, Consistent representation); Accessibility DQ (Accessibility, Access security). The strength of this framework is the breadth of DQ characteristics. Data quality is a multi-dimensional concept dependent on multitude of factors and adoption of data standards does facilitate DQ, but does not guarantee it [29]. Good quality data that meet many of DQ dimensions are critical for public health surveillance purposes. With increasing electronic data exchange and emphasis on interoperability, it is essential to understand impact of various facets of data exchange on various dimensions of DQ.

The Minnesota Electronic Disease Surveillance System (MEDSS) [30] is the public health information system for supporting infectious disease surveillance at a state level for Minnesota and operational since 2008. It holds data on reportable conditions and receives ELRs submitted to the state public health agency. MEDSS is used for case management, contact tracing and to support outbreak investigations. Its scope has expanded to include non-infectious diseases such as blood lead surveillance and birth defects. It’s a person-centric surveillance system which currently holds ~1,279,986 events across infectious diseases, lead and community and family health programs. Approximately 153,880 lab tests/results were reported electronically for 2017 across six health systems and four reference labs. Many healthcare systems are currently on a waiting list for either onboarding/move to electronic exchange or upgrade to better version of reporting standard.

Nationally recommended standards for ELR [4] comprise of HL7 2.5.1 for message format and LOINC [31] and SNOMED [32] codes for representation of lab tests and results respectively. With increasing demands for electronic data exchange for incoming data to MEDSS from clinical sectors and for outgoing data to Centers for Disease Control and Prevention (CDC), new informatics tools to support data validation and exchange were implemented. The objective of this study was to assess the data exchange process and to understand its impact on the quality of data in MEDSS. The overarching goal is to utilize findings for improvements in informatics tools and processes to enhance the value of MEDSS by providing good quality data to support various public health purposes including disease surveillance.

Methods

The process of assessing data exchange for electronic lab reporting and its impact was a mixed methods approach with qualitative data obtained through expert discussions and quantitative data obtained from queries of the MEDSS system. Various subject matter experts (n=9) were identified spanning across the informatics team that supports MEDSS operations, public health program professionals who are users of the MEDSS system and its data, and the Information Technology (IT) team which supports the data exchange process. The focus included both on-boarding (process of shifting to electronic exchange for either new reporting or migration/upgrade to different standard) and on-going submissions. ELR is unique in that reporting can occur from either EHR or from LIMS (Laboratory Information Management System) and can occur from healthcare delivery organization or from reference laboratories and these were taken into consideration. Interviews were conducted in an open-ended/discussion format and were done over time frame of November 2017 through February 2018. Based on these discussions, two high level categories of data exchange process which could impact data quality were identified: onboarding for electronic lab reporting and internal data exchange routing.

Figure 1 displays the ELR onboarding process and includes the testing and validation suite of tools offered in public domain by the National Institute of Standards and Technology (NIST) [33]. The six identified key processes that influence quality of data are numbered A through F (A - mapping of tests and results to appropriate codes, B - NIST test bed for testing of messages, C - submit test HL7 messages, D - solicit HL7 messages with test cases (e.g. specific tests, seasonal diseases), E - technical review, F - program review). Figure 2 displays the internal data exchange routing process which includes the PHIN Messaging System (PHIN MS) [34], a CDC provided software that serves as a transport mechanism for effective movement of messages. This part comprises of four main components numbered G through J (G - PHIN MS, H - Lab code list database validation, I – Rhapsody® Integration Engine [35] rules, J - mapping in MEDSS).

Figure 1

Overview of ELR Onboarding Process

Figure 2

Overview of Internal Data Exchange Routing Process

The potential influence of the ten identified critical steps in the data exchange process and its impact on quality of data was identified through expert input using Strong’s DQ framework as a guidance. This was followed by analysis of data in MEDSS by criteria identified by the informatics team. Evaluation of messages not mapped to any disease program in MEDSS was identified as a priority. Next, assessment of completeness of race and ethnicity fields before and after implementation of demographic data import feature in ELR was completed. Using Influenza reporting as a scenario, the number of non-reportable tests that get submitted and added to data in MEDSS was examined. Finally, the number of incoming messages which get rejected due to errors was examined to quantify the need for additional technical assistance.

Results

The process of exchanging data electronically is iterative and is initiated with numerous rounds of message testing and varying gradation of technical assistance based on data submitter need and capabilities. Each step in the process was deemed critical in its impact on the quality of data which moves across clinical sector and public health. Table 1 lists the six identified key processes for ELR onboarding, relevant sub-processes/notes and their influence including both DQ metric and DQ dimension. All DQ metrics (Intrinsic DQ, Contextual DQ, Representational DQ, and Accessibility DQ) were impacted with varying influence on DQ dimensions. Some errors such as improper mapping on EHR end had a cascading effect and can pass through technical filters and go undetected till use of data by epidemiologists. Some DQ dimensions such as accuracy, relevancy, value-added data and interpretability are more dependent on users at either end of the data exchange spectrum, the relevant clinical groups and the public health program professionals.

Table 1

Onboarding for Electronic Lab Reporting and Data Quality

Data Exchange Process for ELR Onboarding	Data Quality Metric and Data Quality Dimension Impact
A. Mapping of Tests and Results to Appropriate Codes    - Completed in the clinical healthcare space (EHR system and LIMS)	Intrinsic DQ (Accuracy, Objectivity)
B. Test messages using NIST Test Bed    - Ability to map content to HL7 fields    - Capability to submit data in R (required) fields	Contextual DQ (Completeness)
C. Submit HL7 test messages to MEDSS    - Capability to submit data in R (required) fields    - Complete RE (required, but may be empty) and       O (optional) fields	Contextual DQ (Completeness, Value-added data)
D. Solicit HL7 messages with specific tests, seasonal diseases    - Checking for message formats and codes which may not be present in current HL7 test feeds	Contextual DQ (Completeness, Relevancy)
E. Technical review    - HL7 format checks    - Review of LOINC codes    - Review of SNOMED codes    - Review of LOINC-SNOMED pairs    - Mapping of code pairs with appropriate disease	Contextual DQ (Completeness), Representational DQ (Consistent representation, Interpretability)
F. Program Review    - Confirm mapping of code pairs with diseases    - Check for positive and negative test results    - Check for odd messages	Intrinsic DQ (Objectivity), Contextual DQ (Completeness), Representational DQ (Interpretability)

Table 2 lists the six identified key processes related to on-going production submissions using the internal data exchange routing and their influence on data quality. Similar to the on-boarding process, all DQ metrics (Intrinsic DQ, Contextual DQ, Representational DQ, and Accessibility DQ) were impacted with varying influence on DQ dimensions. The three steps labelled H. (Lab Code List Database Validation), I. (Rhapsody Integration Engine Rules) and J. (Mapping in MEDSS) were deemed critical with high level of need for on-going maintenance. Laboratory tests are constantly evolving along with new lab codes (LOINC) and organisms detected (SNOMED) and their combinations to determine disease changing, some processes (H. I. J.) require frequent review. The analysis also revealed the need for collaboration and some processes are dependent on coordination across MEDSS informatics team, information technology (IT) staff and public health program professionals.

Table 2

Data Quality Impact of Internal Data Exchange Routing Process

Internal Data Exchange Routing Process	Influence on Data Quality Metric and Data Quality Dimension
G. PHIN-MS Transport      - Secure messaging platform for transport of messages	Accessibility DQ(Access Security)
H. Lab Code List Database Validation      - Check to ensure that message contains approved code pairs or rules for exemption      - Update codes and code pairs based on new tests and results	Contextual DQ (Completeness), Representational DQ (Consistent representation)
I. Rhapsody Integration Engine Rules      - Fixes format of incoming messages as per rules      - Converts messages into MEDSS accepted format	Representational DQ (Consistent representation)
J. Mapping in MEDSS      - Assignment of messages to diseases	Contextual DQ (Relevancy), Representational DQ (Interpretability)

The results from analysis of data in MEDSS by various criteria identified by the informatics team is presented in Table 3. Evaluation for cases which are not mapped to any disease program and assigned to “other/unknown” category yielded 952 cases. Assessment of messages for these cases noted an absence of LOINC and/or SNOMED codes and their combination pair for disease assignment. Next, the analysis focused on submission of non-reportable respiratory diseases along with reportable conditions (Influenza) due to issues with special lab test panel, and this identified 366 cases. This was followed by evaluating the number of incoming messages which get rejected due to errors and there currently isn’t any process that keeps track of it. The corresponding impact on data quality metrics due to these identified issues are also presented in Table 3. An enhancement was implemented in January 2018 to import demographic data (race, ethnicity) from ELR feeds and this evaluation presented in Table 4. Of the total of 3,651 electronic lab messages received from January through February 2018, data on Race was present in 2,310 messages and 1680 messages received in that time frame had data on Ethnicity. Comparison of this new data with already existing race and ethnicity data in MEDSS obtained through case reporting and follow-up investigations revealed 270 number of messages wherein race from ELR feed was different than one currently recorded in MEDSS.

Table 3

Identified Issues, Data Quality Impact and Correlations with Data Exchange Processes

Identified Issue	# of cases (time frame)	Data Exchange Process	DQ Impact
Non-assignment of Messages to Diseases
Lack of LOINC and/or SNOMED codes	952 (currently)	▪ Testing during ELR onboarding▪ Validation checks with Lab Code List Database	Contextual DQ (Completeness, Value-added data), Representational DQ (Consistent representation)
LOINC – SNOMED pair missing / not mapped	952 (currently)
Submission of non-reportable Diseases
Presence of numerous non-reportable respiratory pathogens (e.g. adeno virus, corona virus)	366 (over 1 year)	▪ Testing during ELR onboarding▪ Screening with Rhapsody integration engine rules	Contextual DQ (Relevancy)
Missing Messages due to Rejections
Rejection of messages due to format and code issues	? approx. few/day (not tracked)	▪ Validation checks with Lab Code List Database▪ Screening with Rhapsody integration engine rules	Contextual DQ (Value-added data)

Table 4

Demographic Data from Electronic Lab Reports and Influence on Data Quality

Data Imported from ELR	Number (Jan – Feb 2018)	Data Quality Enhancement
Race Data	2,310 / 3,651 (63%)	Contextual DQ (Completeness, Value-added data)
Ethnicity Data	1,680 / 3,651 (46%)

Discussion

Federal regulations and incentives have offered the needed momentum towards electronic reporting to public health. But, there are differences in public health measure reporting [36] with ELR lagging behind immunization reporting due to complexities around multitude of labs associated with reportable conditions, slow adoption of recommended codes and multiple entities/professionals involved in exchange such as clinical labs, reference labs, ordering provider, infection control practitioner and disease epidemiologists. Another key factor to consider is that ELR can be generated from EHRs or from laboratory information systems (LIS) in reference labs or in healthcare settings. This study also portrays the need for constant updates to the various validation tools to ensure errors are not being propagated across the data exchange chain.

This research points to the complexity of the data exchange process by illustrating the numerous stakeholders involved and the critical role each one plays in moving towards interoperability. It also pointed to the need for all data exchange partners to be informed of evolution of standards, both message formats (e.g. HL7) and codes (e.g. LOINC, SNOMED). Some of these exchange mechanisms require technical assistance for either submitter (e.g. labs, providers) and the receiver (e.g. public health) or both of them. National projects such as Digital Bridge [37] and APHL Informatics Messaging Services (AIMS) [38] are aimed to assist in data exchange across jurisdictional boundaries in public health. The data exchange process could be set such that messages get rejected if they fail any of the checks, but will require manual intervention by public health or the data reporters to understand quality issues around rejection and fix them.

The study also presents various testing tools (NIST test bed) and validation engines (Rhapsody, lab code list validation database) that help to automate quality checks and monitor various DQ dimensions. Approaches from other public health reporting such as immunizations wherein provider quality reports [39] are generated could be tried in the context of ELR. Likewise open source software tools have been proposed to support data quality checks for both immunization reporting [39] and ELR [23,40]. Implementation and maintenance of these tools require both financial and technical resources. Importantly, there needs to be overarching guidance and support from national organizations such as CDC to ensure standardization and to facilitate sharing of tools/resources across jurisdictions.

The study revealed that data quality is dynamic and on-going oversight is a collaborative effort by MEDSS informatics team, technical and public health program professionals. Overall, maintenance of good data quality in context of ELR needs a multipronged approach with automated tools, data exchange partners education, technical assistance, regular updates of codes/tools, organizational commitment and national guidelines along with support by informaticians/data quality analysts.

This research depicts the details of processes, people and technology and the need for all the parts to align to make an electronic data exchange truly meaningful by providing good quality to data that fits the purpose (public health surveillance in this case). It highlights the benefits of standardization of data exchange processes which can be applied to other public health transactions. Many public health agencies have seen a trend towards centralization of information technology services which adds another layer of complexity to interoperability efforts. It underscores the value of a public health informatician to be part of electronic exchange of data across various sectors (clinical care, labs) and public health. Finally, this study presents a compelling picture of the interoperability endeavor as a team effort and underscores the critical role an informatics team can play in facilitating the data exchange process.

Limitations

The study has some limitations and focus on some dimensions of data quality by Strong et al., is one of them. Some DQ aspects such as accessibility are not integrated with exchange process and hence were excluded. The research emphasis was determined by criteria outlined by MEDSS informatics team, and was limited based on available data during study period. Some metrics were not tracked and certain tool enhancements were implemented recently by IT support team and thus evaluation was limited. Another limitation is that currently a large volume of ELR submitters are reference labs which are not required to collect race and ethnicity data and hence completeness of those data fields through ELR is limited. Some DQ errors are attributed to frequency of upgrade of codes/validation engine that are driven by organizational resources (finances, trained personnel) / institutional priorities and beyond the scope of this study.

Conclusion

With the growing demands for electronic reporting with public health, there is a need to understand the current processes for supporting electronic exchange and their impact on quality of data. This study focused on electronic laboratory reporting to public health and analyzed both onboarding and internal data exchange processes. Insights gathered from this research can be applied to other public health reporting currently (e.g. immunizations) and will be valuable in planning for electronic case reporting in near future. The study has potential implications in promoting data quality along with electronic exchange to support public health surveillance.