A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data.

OBJECTIVE: Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is 'fit' for specific uses.
MATERIALS AND METHODS: DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework's inclusiveness was evaluated against ten published DQ terminologies.
RESULTS: Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies. DISCUSSION: Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data.
CONCLUSION: A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.

Background and Significance

The era of large-scale health data research networks has arrived, along with high expectations that new electronic data sources and analytic methods will answer questions that cannot be examined using traditional controlled clinical trials.1–5 While large administrative claims databases have long been used for retrospective observational studies, limitations have led to heightened interest in health data from other electronic sources, such as electronic health records (EHRs).6–11 Studies using EHR data have enabled investigators to examine the impact of diagnostic and therapeutic interventions in diverse real-world clinical settings.12–18

Comparative effectiveness studies, patient-centered outcomes research, and pragmatic trials using EHR data captured during routine clinical care from one or more practice settings are becoming an important complement to prospective randomized trials for generating new insights and knowledge. National and international EHR-based clinical research networks are expanding the scope and depth of available data to answer critical questions about care decisions and outcomes important to patients and families.19–23

Detailed clinical data are mostly generated by electronic transactions in operational systems that are not primarily intended for research and secondary analysis. Secondary data use refers to the use of data for purposes other than those for which it originally was collected, such as operational, quality improvement, and research analytics. Access to large quantities of clinical data from operational EHRs holds much promise. However, a major concern is that data not collected systematically for research will be poorer quality, which could have negative impacts on findings generated from these data.24–27 Transaction-oriented systems rarely include prespecified, unambiguous data definitions, and uniform (unbiased) data collection procedures. A substantial body of research suggests that data collected in EHRs and other operational systems may not be of sufficient quality for research.28–37 EHRs typically are optimized for efficient patient care and nonclinical administrative requirements, resulting in great variation in clinical documentation practices, even among users of the same systems.38–40 As clinical data warehouses and large-scale EHR-based data networks become established repositories of electronic health data, consistent methods for describing, assessing, and reporting data quality (DQ) findings could be one way to help secondary data users and consumers understand the potential impact of DQ on reusing data and interpreting findings.36,41

The Need for a Harmonized Data Quality (DQ) Terminology

The current DQ literature is inconsistent in the use of terms that describe the complex multidimensional aspects of DQ.42–51 Inconsistent use of terms to describe DQ features makes it difficult to understand when similar or different DQ features are being discussed. The lack of consistent DQ definitions also makes it difficult to compare DQ results across multiple data-sharing partners. The underlying premise of this work is that standardizing the terms and definitions of DQ concepts, the methods used to evaluate these concepts, and the metrics and formats used to report DQ findings could improve understanding and transparency about the limitations of the data and the results based on these data. The current project focuses on the first challenge—developing a unified DQ terminology and definitions. Our DQ terminology was explicitly scoped to focus on the broad-based evaluation of a large data set typically found in clinical data sharing networks—features that would allow a network to determine the acceptability of data from a contributing partner.

Materials and Methods

Figure 1 depicts a timeline documenting the events that took place as a part of a community-based DQ terminology harmonization effort. Investigators currently engaged in DQ-related work as well as all members of a panel of experts who participated in an earlier effort on DQ reporting sponsored by the Electronic Data Methods (EDM) Forum52 were enlisted to provide expertise on identifying existing DQ models, terms, and assessment methods.

Figure 1.

Timeline of Significant Events in Developing the Harmonized DQ Terminology

To obtain a broad representation of DQ terms in current use, we used materials from the previous EDM Forum project,41 such as the Mini-Sentinel (MS) data characterization routines,53 and DQ rules embedded in the Observational Medical Outcomes (OMOP) and the Observational Health Data Sciences and Informatics (OHDSI) open-source DQ tools.54,55 We also included standard operating procedures for DQ assessment used in past or current projects, published “best practices,” and DQ publications from both the clinical research and information sciences literature.36,43,47,56–60 In total, representatives from approximately 20 of the largest United States distributed research networks or large data owners, along with international engagement, either participated directly or were represented in the assessment materials. Collectively, participants represent clinical data networks that contain data on over 540 million patient records. Based on iterative discussions, an initial draft set of DQ terms, categories, and definitions was developed over nine months and was revised as detailed below.

In July 2014, the EDM Forum hosted two one-day workshops to review and critique the draft DQ terms and definitions. One workshop enlisted participants representing patients, patient advocacy groups, and policymakers interested in DQ. The second workshop enlisted members of the informatics and comparative effectiveness research (CER) community. Both workshops explored the need for clarity and transparency in DQ concepts to the specific community and participants. The patients and policymaker workshop also focused on preferred methods for communicating DQ findings and engagement methods to ensure that DQ efforts include stakeholder perspectives. The informatics and CER community workshop participants reviewed and critiqued the draft DQ terms and definitions. Significant portions of the discussions during each workshop were recorded and were later transcribed, imported into ATLAS.ti (qualitative data analysis software), and reviewed to identify themes and subthemes expressed by participants during each workshop. Recommended changes to the DQ terms, categories, and definitions identified by workshop participants were incorporated into the next version of the terminology. A second round of iterative, expert consensus development was performed by incorporating comments received from a wiki page dedicated to the DQ terminology, as well as introducing the revised harmonized draft terminology presented by webinars to national audiences.

It is estimated that through all outreach efforts, approximately 100 unique individuals from diverse disciplines, as well as United States and international networks and projects, contributed to the development and review of the harmonized DQ terminology. Recommendations that had strong consensus were integrated throughout the process. DQ terms that addressed similar issues were merged and grouped into categories and subcategories. Terms that focused on DQ features intrinsic to data elements—such as their presence or absence, format, values, and distributions—were included in the terminology. Terms that focused on DQ features extrinsic to data elements, such as data access, system availability, security and privacy concerns, and features for determining fitness for a specific analysis (fitness for use) were not included. We examine this scoping decision in the Discussion section. The final set of DQ terms developed by the primary authors were organized into a logical framework with three major categories, which were further separated into two evaluation contexts, Verification and Validation.

Results

Table 1 presents the harmonized set of DQ terms, definitions, and examples organized by categories and subcategories. Each DQ category conveys a DQ concept that needs to be interpreted within a certain context. Contexts distinguish different strategies for assessing DQ. Therefore, all DQ categories are divided into Verification and Validation (Table 1, top row). The key difference between these two DQ assessment contexts is framing expectations. Verification focuses on how data values match expectations with respect to metadata constraints, system assumptions, and local knowledge. This DQ context does not rely on an external reference or benchmark. Expectations for DQ measures are internally derived, based on expert judgment, relevant heuristics, and knowledge of impossibilities or contradictions. DQ assessment measures in this context can be created using information available within the existing data environment. While the scope of “relevant heuristics and local knowledge” may vary across environments (e.g., one group may have access to domain experts not available to another group), the key feature with Verification is the ability to determine expected values and distributions using resources within the local environment.

Table 1.

Harmonized DQ Terms, Definitions, and Examples: Organized by Verification and Validation Contexts Within Categories and Subcategories

VERIFICATION		VALIDATION
DEFINITION	EXAMPLE	DEFINITION	EXAMPLE
CONFORMANCE: DO DATA VALUES ADHERE TO SPECIFIED STANDARDS AND FORMATS?
VALUE CONFORMANCE
a. Data values conform to internal formatting constraints.b. Data values conform to allowable values or ranges.	a. Sex is only one ASCII character.b. Sex only has values “M,” “F,” or “U.”	a. Data values conform to representational constraints based on external standards.	a. Values for primary language conform to ISO standards.
RELATIONAL CONFORMANCE
a. Data values conform to relational constraints.b. Unique (key) data values are not duplicated.c. Changes to the data model or data model versioning.	a. Patient medical record number links to other tables as required.b. A medical record number is assigned to a single patient.c. Version 1 data does not include medical discharge hour.	a. Data values conform to relational constraints based on external standards.	a. Data values conform to all not-NULL requirements in a common multi-institutional data exchange format.
COMPUTATIONAL CONFORMANCE
a. Computed values conform to computational or programming specifications.	a. Database- and hard-calculated Body Mass Index (BMI) values are identical.	a. Computed results based on published algorithms yield values that match validation values provided by external source.	a. Computed BMI percentiles yield identical values compared to test results and values provided by the CDC.
COMPLETENESS: ARE DATA VALUES PRESENT?
a. The absence of data values at a single moment in time agrees with local or common expectations.b. The absence of data values measured over time agrees with local or common expectations.	a. The encounter ID variable has missing values.b. Gender should not be null.c. Medical discharge time is missing for three consecutive days.	a. The absence of data values at a single moment in time agrees with trusted reference standards or external knowledge.b. The absence of data values measured over time agrees with trusted reference standards or external knowledge.	a. The current encounter ID variable is missing twice as many values as the institutionally validated database.b. A drop in ICD-9CM codes matches implementation of ICD-10CM
PLAUSIBILITY: ARE DATA VALUES BELIEVABLE?
UNIQUENESS PLAUSIBILITY
a. Data values that identify a single object are not duplicated.	a. Patients from a single institution do not have multiple medical record numbers.	a. Data values that identify a single object in an external source are not duplicated.	a. An institution’s CMS facility identifier does not refer to a multiple institutions.
ATEMPORAL PLAUSIBILITY
a. Data values and distributions agree with an internal measurement or local knowledge.b. Data values and distributions for independent measurements of the same fact are in agreement.c. Logical constraints between values agree with local or common knowledge (includes “expected” missingness).d. Values of repeated measurement of the same fact show expected variability.	a. Height and weight values are positive.a. Counts of unique patients by diagnoses are as expecteda. Distribution of encounters per patient or medications per encounter distributions are as expectedb. Serum glucose measurement is similar to finger stick glucose measurement.b. Oral and axillary temperatures are similar.c. Sex values agree with sex-specific contexts (pregnancy, prostate cancer).d. Height values are similar when taken by two separate nurses within the same facility using the same equipment.	a. Data values and distributions (including subgroup distributions) agree with trusted reference standards or external knowledge.b. Similar values for identical measurements are obtained from two independent databases representing the same observations with equal credibility.c. Two dependent databases (e.g., database 1 abstracted from database 2) yield similar values for identical variables.	a. HbA1c values from hospital and national reference lab are statistically similar under the same conditions.a. Distribution of patients with cardiovascular disease diagnoses are similar to CDC rates for the same age and sex groupsa. Readmission rates by age groups for Medicare patients agree with CMS valuesb. Diabetes ICD-9CM and CPT codes are similar between two independent claims databases serving similar populations.c. Recorded date of birth is consistent between EHR data and registry data for the same patient.
TEMPORAL PLAUSIBILITY
a. Observed or derived values conform to expected temporal properties.b. Sequences of values that represent state transitions conform to expected properties.c. Measures of data value density against a time-oriented denominator are expected based on internal knowledge.	a. Admission date occurs before discharge date.b. Date of an initial immunization precedes date of a booster immunization.c. Similar counts of patient observations between extraction-transformation-load cycles.c. Counts of emergency room visits by month shows expected spike during flu season.c. Medications per patient-day are as expected	a. Observed or derived values have similar temporal properties across one or more external comparators or gold standards.b. Sequences of values that represent state transitions are similar to external comparators or gold standards.c. Measures of data value density against a time-oriented denominator are expected based on external knowledge.	a. Length of stay by outpatient procedure types conforms to Medicare data for similar populations.b. Immunization sequences match the CDC recommendations.c. Counts of emergency room visits by month shows spike during flu season that are similar to local health department reports.c. Medications per patient-day matches claims data.

Notes: The lettering in each column can be used to map each definition to its corresponding example. Not every definition has a corresponding example.

Extract, Transform, Load ETL (ETL); International Organization for Standardization (ISO); Electronic Health Record (EHR) Data; International Classification of Diseases, Ninth and Tenth Revisions (ICD-9CM and ICD-10CM); Current Procedural Terminology (CPT); Centers for Medicare & Medicaid Services (CMS); Centers for Disease Control and Prevention (CDC).

Validation focuses on the alignment of data values with respect to relevant external benchmarks. In Validation, expectations are derived from comparisons to known true or relative gold standards and external knowledge that exists as resources independent of the data source being evaluated. Declaring a resource or benchmark to be a gold standard requires special knowledge about the degree of confidence or trust that a community of users has in an external data source.26 Another common method of creating a relative gold standard can be to combine results across multiple data providers, such as all data partners in a data network, to create a merged data set that can be used as an “external” comparator.

Verification and validation can be equally appropriate to assess EHR DQ, depending on the data context and intended use. EHR data is a novel source of information in many organizations and communities. As a result, verification of local data may be the only appropriate reference against which to frame expectations and outcomes. For example, EHR data may be used to generate estimates on a denominated population for which a near-census is available. In such cases, validation against national survey data is likely to be a poor reference due to fundamental differences in the goals of data collection and subsequent impacts on survey design decisions such as the sampling frame and the granularity of data elements collected. Context and intended use of EHR data motivate incorporating a verification or validation assessment strategy (or both).

DQ Category 1: Conformance

Conformance focuses on DQ features that describe the compliance of the representation of data against internal or external formatting, relational, or computational definitions. Conformance DQ measures do not attempt to assess the completeness or plausibility of the values that are recorded but merely whether the values that are present meet syntactic or structural constraints. Expected conformance features are often described in a document called a “data dictionary,” which lists the intended format and allowed values for every data element. Conformance DQ features are divided into three subcategories: value conformance, relational conformance, and computational conformance.

Value Conformance

Value conformance seeks to determine if recorded data elements are in agreement with a prespecified, constraint-driven data architecture. Internal data constraints are typically imposed by a formal data model, which specifies expectations for data types, data domains and allowed values, and data formats. These constraints typically are documented in a data dictionary. Validation adds constraints imposed by conforming to external standards for data representation and values, such as value set constraints imposed by external terminology standards.

Relational Conformance

Relational conformance seeks to determine if the recorded data elements are in agreement with additional structural constraints imposed by the physical database structures that store data values. In this category are conformance to nullability constraints (data fields that are allowed to null or must always have a value) and to primary key and foreign key relationships. Both Verification and Validation relational constraints express how a data model or standards body represents reality; the structures that define these constraints are usually represented in metadata descriptions or implemented as database integrity rules or attribute domain sets.

Computational Conformance

Computational conformance seeks to determine if computations used to create derived values from existing variables yield the intended results either within a data set (Verification) or between data sets (Validation), when programs are based on identical specifications. Computational conformance focuses on the correctness of the output value of calculations against technical functional specifications. Other DQ categories, such as Plausibility, focus on the tenability of the calculated results. That is, a calculation may be correct according to its formal specification (Computational Conformance) yet not be an accurate representation of the intended concept (Plausibility) because the specification is incorrect or incomplete. Areas of concern in this category are the use of correct logic and formulas, including conditions with unusual or unexpected inputs, which yield the intended output under all circumstances.

DQ Category 2: Completeness

Completeness focuses on features that describe the frequencies of data attributes present in a data set without reference to data values. Completeness measures assess the absence of data at a single moment over time or when measured at multiple moments over time, without reference to its structure or plausibility, which are assessed in the Conformance and Plausibility DQ categories respectively. Not explicitly represented by a separate Completeness subcategory is the critical concept of why data are complete or incomplete (i.e., the underlying mechanism leading to incompleteness), commonly known as “missingness” in the statistics literature.61,62 While there are important differences between types of missingness (i.e., missing at random, missing completely at random, and missing not at random), these differences do not require different conceptualizations within the DQ category of Completeness, but do suggest different assessment methods, analytic approaches, and underlying root causes that lead to incompleteness.62,63

The restriction of completeness to the presences of data “without reference to data values” is an important feature that distinguishes counts that are considered measures of completeness from counts that are considered measures of plausibility.

DQ Category 3: Plausibility

Plausibility focuses on features that describe the believability or truthfulness of data values. For this category, plausibility is determined by a variable’s value, when a value is placed within the context of another variable (i.e., two independent variables assessing the same construct), or a temporal sequence or state transition (i.e., patient follow-up treatment for a disease must be preceded by a corresponding diagnosis). Unlike Conformance and Completeness, which focus only on the structure and presence of values respectively, Plausibility focuses on actual values as a representation of a real-world object or conceptual construct by examining the distribution and density of values or by comparing multiple values that have an expected relationship to each other.

The term “plausibility” was carefully selected; it implies the existence of an acceptable variable value range and distribution rather than requiring the existence of a single absolute truth. Alternative words, such as “accuracy,” “correctness,” and “validity” were considered and set aside because these words have a wide range of competing and inconsistent definitions in the literature and are applied differently in the psychometric and DQ communities. Plausibility has uniqueness, atemporal, and temporal subcategories.

Uniqueness Plausibility

The Uniqueness subcategory seeks to determine if objects (entities, observations, facts) appear multiple times in settings where they should not be duplicated or cannot be distinguished within a database (Verification) or when compared with an external reference (Validation). Duplication frequently occurs when disparate data streams that contain overlapping objects are combined. Data extraction errors, such as incomplete relational join conditions, can also generate duplicate records.

Atemporal Plausibility

Atemporal Plausibility seeks to determine if observed data values, distributions, or densities agree with local or “common” knowledge (Verification) or from comparisons with external sources that are deemed to be trusted or relative gold standards (Validation).26 Examples include physical quantities that cannot be negative, exceed possible boundaries, or (in health care) represent physiologically impossible states. Expected values, distributions, or densities may vary based on context; variables may have different expected distributions or densities when stratified by age, gender, and socioeconomic values.

In settings where logic or knowledge do not provide clear guidance on expectations, external gold standards created by organizations thought to implement strict DQ standards can represent an external reference source of comparisons for variable values and distributions. Atemporal plausibility also focuses on the observed relationships between independent variables that have expected or known relationships, either due to local knowledge about how concepts in the data set should be related or due to physical or conceptual constraints that should tie variables together in a specific manner. Included in this subcategory is the examining of whether repeated values of the same observation or fact align as expected. The expected variability of repeated measurements of the same fact (observation, event) comes either from local knowledge or from comparisons with measurements drawn from similar external data sources.

Temporal Plausibility

Temporal plausibility seeks to determine if time-varying variables change values as expected based on known temporal properties or across one or more external comparators or gold standards. Temporal properties that establish expectations in this subcategory include temporal stability (do values vary over time as expected), temporal continuity (do values persist over time as expected), state transitions (do sequences of events occur as expected), and temporal dependencies between time-varying variables. In health care data, additional observed temporal properties may be temporal cycles, such as diurnal variations or cyclical treatment regimens, and temporal reoccurrences, such as recurring disease flares. Computations that derive temporal abstractions, such as condition durations or medication exposure intervals based on medication ordering events, would not be included in this category but would be included in the computational conformance DQ category, even though time is involved in calculating the derived temporal concepts.

Table 2 illustrates the alignment of the harmonized DQ terms, categories, and subcategories with 10 previously published terminologies. When comparing the subset of terms and categories within these terminologies that apply to intrinsic DQ features and not to external operational features such as data access and security, the harmonized terminology had good coverage. A majority of the existing terminologies adequately mapped to DQ concepts within the data verification context, and fewer mapped to DQ concepts within the data validation context.

Discussion

The DQ framework and terminology presented in Table 1 aligns a wide array of existing DQ concepts that have had different and sometimes inconsistent definitions. In a nonmedical context, Wang and Strong enumerated 179 different DQ terms (called “DQ attributes”) collected as part of a survey with business data consumers.42 From a review of the electronic medical record literature, Weiskopf and Weng similarly categorized 27 unique terms for DQ dimensions.45 Other approaches, such as the ontological methods by Liaw46 and Johnson,51 have been used to develop alternative DQ terms. In the current work, community-based consensus, iterative refinement, and continuous alignment across existing DQ terminologies were used to arrive at a common set of consensus terms and definitions organized into a three-category framework.

Our framework highlights two distinct data-quality assessment contexts that distinguish two distinct strategies for the source of expectations or comparisons of EHR data based on internal characteristics (Verification) or external resources (Validation). We make no claim that one strategy or set of expectations is “better than” another source. In fact, we argue that a comprehensive assessment program will include both types of assessments to address these equally important but distinct aspects of DQ.

The ordering of the DQ categories (Conformance, followed by Completeness, followed by Plausibility) in Table 1 is purposeful. The ordering represents the additive effect of a category on the next category. Conformance focuses strictly on the agreement of values against various technical specifications without regard to the amount or believability of those values. Completeness focuses on the absence of data of a variable, again without regard to the believability of those values. Finally, Plausibility focuses on the believability or correctness of data and counts of data that conform to technical specifications (Conformance) and are present (Completeness) in the data. The restriction of completeness to the presence of data “without reference to data values” is an important feature that distinguishes completeness measures from plausibility measures. Counts (or density) by values (or by grouping) is based on expected distributions by observed values. We placed all DQ features that are dependent on values (or groupings of values such as dates by quarter) into Plausibility. That is, measures that must be interpreted in the presence of data values are categorized as assessing the believability of the distribution across values.

Our harmonized terminology and framework focuses exclusively on DQ issues related to the intrinsic features of data values. Wang and Strong defined intrinsic DQ as DQ features that involved only the data values “in their own right” without reference to external requirements or tasks.42 They differentiated intrinsic DQ features from contextual DQ features, which typically entail unique contextual or task-specific DQ requirements. Other terminologies in Table 2 include DQ issues related to operational features such as data access, availability, security, maintainability, and fitness-for-use features—such as timeliness, appropriate amount for intended use, and relevancy. These extrinsic DQ concepts are not represented in the harmonized terminology but will be the focus of future work.

The DQ literature uses the concept of fitness for use or fitness for purpose, where knowing the intended use of data determine when DQ assessment results are declared as fit or not fit.43,66 Fitness for use does not change the underlying intrinsic DQ features of the elements in a data set; it does change the acceptability of measures of DQ based on the intended use. For example, a completeness measure of 70 percent may be acceptable for a variable that is known to be not relevant to an analysis but would be unacceptable in an analysis where the variable was deemed important.

Our DQ terminology does not include terms such as accuracy, validity, or correctness that are commonly found in other DQ terminologies (Table 2). These terms are used in widely differing ways across the DQ literature. In addition, they have very specific meanings in the psychometric methods development community that are different than their intended meaning in a DQ context. Given the large diversity of interpretations and the competing use of the same terms in a closely related field that caused significant confusion while constructing our harmonized terminology, we avoided using these terms—while selecting terms that captured our intended concept that represents the trueness of data values.67,68 A secondary goal was to select terms that are easily assessable to a nontechnical audience and that have the least likelihood of being confused with alternative existing technical uses of the same word.

DQ issues can occur in data sets constructed from a single institution. More challenging is understanding DQ issues in data sets constructed by combining data sets from multiple separate institutions. Differences in how data are captured and stored in the original source-data system—usually the institutional EHR or enterprise data warehouse, how data are extracted and transformed into the analytic data, and the impact of data workflows or provenance can cause significant challenges in ensuring common data formats (syntax) and meaning (semantics).69 Currently, most data networks rely on manual processes for establishing common data definitions and reproducible data extraction conventions. Hence, DQ issues can appear in any data source, even those generated by highly skilled technical staff. The use of a harmonized DQ terminology will not avoid DQ issues but is the first step of a collective community-based effort to identify and harmonize their use in a consistent manner.

Limitations

While the broad and diverse group of collaborators who developed the harmonized DQ terminology represent diverse networks that include hundreds of millions of patients, feedback regarding the harmonization terminology was generated from a convenience sample of EDM Forum members. However, a wide range of outreach methods were used to ensure representative community stakeholder engagement.70 Most of the researchers who developed DQ terminologies included in Table 2 were collaborators and co-authors. While over 30 individuals provided input during the development of the terminology, many other relevant constituencies were not included, such as individuals responsible for using clinical data for operational and quality improvement activities within health systems, technical staff developing enterprise data warehouses or data marts, and journal editors interested in ensuring DQ transparency in their publications.

As highlighted in Table 2, the scope of this harmonization effort was limited to intrinsic DQ issues that focus on the structure and presence of data and their values. Not included in this terminology are other widely identified dimensions of DQ that focus on operational or fitness for use features. These DQ concepts are important in the context of a specific data system and environment, and the unique criteria that determine if a data set can answer a specific question. In addition, we did not include DQ issues that arise from deficiencies in the data representation or data model used to store data values. For example, a data model that captures International Classification of Diseases, Ninth and Tenth Revisions (ICD-9CM and ICD-10CM) codes for diagnoses in a single table but does not allow for representing data provenance is unable to distinguish between admitting, preliminary, and discharge diagnoses. This limitation of the data model can have a significant impact on the fitness for use of a data set but is not considered part of the DQ framework presented here. We discuss data model issues and its impact on CER elsewhere.71

Similarly, our framework does not focus on the quality of the definition of a data variable other than how that definition has an impact on expectations about values associated with that variable. Inadequacies or inaccuracies in the definition of a variable may result in high-quality values that do not represent the intended concept and therefore are not useful.

This analysis focused on “traditional” data extracted from EHRs and administrative systems. In the era of big data, which brings together new data sources with widely varying data characteristics, new DQ concepts, measures, and computational methods could emerge, resulting in expansion or alteration of the current terminology. For example, biological “-omics” data may generate new DQ concepts based on unique DQ features seen in these data that are not present in clinical and EHR data. Similarly, real-time physiological signals, wearable medical devices, and patient-entered data may also have unique DQ features that require additional terms. Methods are needed for ensuring that this terminology remains all-inclusive while being continually adapted to accommodate new uncovered DQ dimensions. A table has been posted on a wiki hosted by the EDM Forum (http://repository.academyhealth.org/dqc/) where new DQ terms and methods can be debated and added by the community.

Finally, the terminology in Table 1 represents only one part of a comprehensive unified DQ framework, which would include DQ assessment methods and DQ reporting standards. Proposed DQ reporting standards have been published previously.52

Future Work

Table 2 validates the inclusive breadth of our harmonized DQ concepts and framework against a wide range of published DQ terms. A next step in ensuring that the harmonized DQ concepts are comprehensive is to evaluate existing DQ assessment measures (also called “DQ checks”) to ensure that these DQ measures have a corresponding DQ category and term. An assessment method that cannot be unambiguously assigned to a DQ concept implies that either the concepts are not clearly defined or may indicate a missing concept. One difficulty in performing this assessment is that many DQ activities are performed as study-specific “one-off” analyses that are not well documented and are therefore difficult to discover. Some large national networks have published their DQ checks or have made their DQ check code available as open source. MS53 and OHDSI Achilles Heel are examples.72

The ultimate value of this work will be seen when data owners develop and report DQ measures based on the DQ categories in Table 1. To encourage adoption of these measures, we are developing a DQ assessment toolkit that will compute standardized DQ measures and data visualizations that will help identify DQ issues using open source tools. Critical to this work is the development of a data model for storing DQ measures that is not tied to the specific data formats used to store the original data set.

In a previous publication, we provide recommendations for reporting DQ results.52 However, technical and nontechnical barriers to reporting on DQ issues may prevent investigators from sharing DQ results. For example, concerns about reputational or organizational impact from revealing DQ issues could lead to reluctance to make these findings widely available. We have initiated a community survey to explore these possibilities.

Conclusions

The arrival of the era of larger volumes of electronically available data has increased the availability and reuse of EHR data. While these data have great potential for significant advancement in clinical practice and research, the quality of these data sources ultimately determine their utility. To fully understand and accurately characterize the limitations of these data sources, establishing standardized, validated methodologies for assessing and reporting DQ is crucial. The current project developed a comprehensive DQ terminology organized into a three-tier conceptual DQ framework for EHR data. Future research on the DQ terminology should aim to verify its generalizability and utility, add fitness for use terms and concepts, and extend the application of the terminology to emerging forms of digital health data poised to provide new insights into health and well-being. Future efforts should also focus on expanding this work from a harmonized terminology to a harmonized operational framework that includes reusable DQ assessment, visualization, and reporting capabilities for understanding the strengths and limitations of EHR data for secondary use.

Table 2.

Crosswalk Between Harmonization Terminology, Categories, and Subcategories Versus Pre-Existing Categories and Frameworks

PROPOSED CATEGORIES	JOHNSON 2015 51a	ZOZUS 2014 50b	LIAW 2013 46c	WEISKOPF 2013a 48d	WEISKOPF 2013b 45e
CONFORMANCE
Value	Representation-Integrity Coding-Consistency Representation-Consistency	Consistency	Internal Consistency External Consistency		Plausibility
Relational	Domain-Consistency Domain-Metadata	Data Element Completeness Information Loss and Degradation
Computational			Correctness (Accuracy Elements)		Concordance
COMPLETENESS
Completeness	Representation-Complete Domain-Complete Relative-Completeness	“Column” Data Value Completeness	Completeness (Elements of Correctness)	Documentation Completeness Density Completeness	Completeness
PLAUSIBILITY
Uniqueness		Ascertainment Completeness	No Duplication
Atemporal	Domain-Consistency Relative- Correctness Relative-Completeness	Representational Inaccuracy Information Loss and Degradation Consistency	Correctness (Reliability Elements) Consistency (Reliability Elements) External Consistency	Density Completeness	Correctness Concordance Plausibility
Temporal	Representation-Correctness	Consistency

Notes: Existing DQ approaches were organized chronologically; approaches with the same year of publication were ordered alphabetically. Only the first author and publication date are provided in the table.

Johnson51 Correctness (RepresentationIntegrity, RelativeCorrectness, RepresentationCorrectness, Reliability); Consistency (RepresentationConsistency, DomainConsistency, CodingConsistency, DomainMetadata); Completeness (RepresentationComplete, DomainComplete, RelativeCompleteness, Sufficiency, DomainCoverage, TaskCoverage, Flexibility, Relevance); Currency (RepresentationCurrent, Dataset Current, TaskCurrency). The proposed terminology does not capture Reliability, Sufficiency, DomainCoverage, TaskCoverage, Flexibility, Relevance, RepresentationCurrent, DatasetCurrent, or TaskCurrency.

Zozus50 Completeness (Data Element Completeness, “Column” Data Value Completeness, “Row” Data Value Completeness, and Ascertainment Completeness); Accuracy (Representational Inadequacy, Information Loss and Degradation); Consistency has no lower-level terminology. The proposed terminology does not capture “Row” Data Value Completeness.

Liaw46 The proposed terminology does not capture: Timeliness, Relevance, Usability, or Security.

Weiskopf48 Completeness (Documentation, Breadth, Density, and Prediction). The proposed terminology does not capture Breadth or Prediction.

Weiskopf45 Both Plausibility and Concordance are proxies of Correctness. The proposed terminology does not capture Currency.

PROPOSED CATEGORIES	KAHN 2012 44f	NAHM 2012 57g	MCGILVRAY 2008 64h	EPPLER 2006 65i	WANG 1996 42j
CONFORMANCE
Value	Attribute Domain Constraints Historical Data Rules State-Dependent Object Rules	Granularity Precision	Data Integrity Fundamentals		Representational Consistency
Relational	Relational Integrity Rules	Attribution	Data Specifications
Computational	Attribute Dependency Rules
COMPLETENESS
Completeness	Attribute Domain Constraints	Completeness	Data Integrity Fundamentals		Completeness
PLAUSIBILITY
Uniqueness	Relational Integrity Rules		Duplication
Atemporal	Attribute Domain Constraints Relational Integrity Rules Attribute Dependency Rules	Consistency (Internal) Granularity	Data Integrity Fundamentals Accuracy Consistency and Synchronization	Consistency Correctness Accuracy	Believability Accuracy Representational Consistency
Temporal	Historical Data Rules Attribute Dependency Rules State-Dependent Object Rules	Accuracy	Data Integrity Fundamentals

Notes: Existing DQ approaches were organized chronologically; approaches with the same year of publication were ordered alphabetically. Only the first author and publication date are provided in the table.

Kahn44 Attribute Domain Constraints (Attribute Profiling, Optionality, Format, Valid Valúes, Precisión); Relational Integrity Rules (Identity, Reference, Cardinality, Inheritance); Historical Data Rules (Currency, Retention, Granularity, Continuity, Timeline Patterns, Value Patterns, Event Dependencies, Event Conditions, Event Attributes); State-Dependent Object Rules (State-Transition Profiling, State Domain, Action Domain, Terminator Domain, State-Actions); Attribute Dependency Rules (Continuity, Duration, Redundant Attributes, Derived Attributes, Partially Dependent Attributes, Conditional Optionality, Correlated Attributes). The proposed terminology does not capture Historical Data Rules: Retention.

Nahm57 Inherent (Accuracy, Currency, Completeness, Consistency (internal), Specificity, Attribution); Context Dependent (Timeliness, Relevance, Granularity, Precisión). The proposed terminology does not capture Currency, Timeliness, Relevance, or Specificity.

McGilvray64 The proposed terminology does not capture Timeliness And Availability, Ease Of Use And Maintainability, Data Coverage, Presentation Quality, Perception, Relevance and Trust, Data Decay, or Transactability.

Eppler65 Community Level (Comprehensiveness, Accuracy, Clarity, Applicability); Product Level (Conciseness, Consistency, Correctness, Currency); Process Level (Convenience, Timeliness, Traceability, Interactivity); Infrastructure Level (Accessibility, Security, Maintainability, Speed).The proposed terminology does not capture Comprehensiveness, Clarity, Applicability, Conciseness, Currency, Convenience, Timeliness, Interactivity, Accessibility, Security, Maintainability, or Speed.

Wang42 Intrinsic DQ (Believability, Accuracy, Objectivity, Reputation); Contextual DQ (Value-Added, Relevance, Timeliness, Completeness, Appropriate Amount of Data); Accessibility DQ (Accessibility, Access Security); Representational DQ (Interpretability, Ease of Understanding, Representational Consistency, Concise). The proposed terminology does not capture Value-Added, Cost-effectiveness, Relevancy, Interpretability, Ease of Understanding, Ease of Operations, Accessibility, Flexibility, Objectivity, Timeliness, Reputation, Concise, Access Security, Appropriate Amount of Data, Variety of Data, or Traceability.