Electronic health data quality maturity model for medical device evaluations.

Challenges in the conduct of medical device evaluations are well established, a key impediment being a lack of investment in registries and other enabling approaches.1 The increasing availability of real world data (RWD), defined as data generated in the course of clinical care, or the activities of daily life, and the application of appropriate analytical and statistical methods to RWD, have increased the opportunities to improve the quality and speed, while decreasing the cost, of generating evidence about medical products.2

RWD can be derived from electronic health record (EHR) systems, clinical documentation, claims, patients, devices, sensors and wearables. In USA, EHR systems have now been adopted by over 96% of non-federal acute care hospitals, stimulated by the American Recovery and Reinvestment Act of 2009.3 Similar trends are occurring around the world, with the global EHR market exceeding US$30 billion in 2018.4 However, there are a number of known issues related to the quality and suitability of EHR data for research purposes. First, despite advances in information modeling and interoperability, EHR data captured at the point of care do not generally meet research-grade standards. Medical device data are typically unstructured, with clinical documentation regarding procedures still being documented primarily as verbose (text) rather than as structured data. Information such as clinical descriptors, complications, and outcomes are rarely captured as discrete, auditable data. Common data models (CDMs) such as Sentinel and PCORnet provide a framework for organizing and transforming data into consistent representations and are now being used to provide Real World Evidence (RWE) for research purposes.5 6 Second, EHR data may be incomplete. For example, while the regulations regarding the implementation of a unique device identifier (UDI) are at an advanced stage, integration of the UDI into care processes (and documentation) by healthcare systems is lagging substantially.7 Finally, the ability to follow an individual patient over an appropriate timeframe requires access to data across the various institutions where the patient has received care. Because of the ability of insurance claims data to follow patients across care settings, linkages between EHR data and claims data have been used for these purposes. This aggregation, however, requires complex governance and data-sharing agreements. In most cases, study-specific data will still be required to answer specific research questions through linkage with other existing data sources such as health system registries, supply chain information or manufacturer registries.8

As regulators and other decision makers continue to receive evidence that uses RWD, the ability to understand and evaluate the quality of the data in a given study will be critical. The contribution by Drozda et al in this edition of BMJ Surgery, Interventions and Health Technologies 9 highlights the work to create a source of RWE via RWD from EHR systems that supports the evaluation of medical devices. The strategy of the Mercy Health System (St. Louis, Missouri, USA) included the standardization of clinical terminology based on professional society registries, capture of well-defined granular clinical data at the point of care, integration of the UDI into care processes, the aggregation of longitudinal data reflecting computable outcomes (phenotypes), the incorporation of claims data, and formalization of a data model to serve analytics purposes. As such, the Mercy approach provides an excellent schema for other interested organizations as well as the scientific and clinical communities.

How can other healthcare systems develop such systems? What can be gleaned from this experience for other healthcare organizations seeking to develop a similar infrastructure? We believe that the road map starts with the development of a Data Quality Maturity Model. The model will provide a framework for institutions to self-evaluate, report their status on a number of key items related to the quality of the institution’s RWD and make investments to advance their data systems.

One such model is proposed by the National Evaluation System for health Technology (NEST) program created by the US Food and Drug Administration.10 The NEST Coordinating Center (NESTcc) Data Quality committee proposed a maturity model with a goal to describe a framework for assessing and advancing the competencies of an organization within a domain (eg, healthcare technologies). Organizational maturity is the demonstration of operational capability and capacity within a specific domain. The typical components described in the levels of a maturity model are organizational governance and culture, processes, and/or technology tools, and systems.11 Maturity models are applied by organizations to assess current levels of competencies, model business capabilities, and prioritize future investments. A well-known maturity model for business data governance has been described by Stanford University. This model addresses the dimensions of people, policies, and capabilities across the attributes of awareness, formalization, metadata, stewardship, data quality, and master data.12 Examples of published healthcare maturity models include the Healthcare Information and Management Systems Society (HIMSS) EHR Adoption Model13 and the HIMSS Adoption Model for Analytics Maturity.14

Key data process domains that drive data quality

Optimally, use of healthcare system RWD requires competency across several data process domains, including data consistency, completeness, and automation.15 A foundational requirement is consistency of clinical data based on standardized data dictionaries and/or applicable data standards. While data consistency can be most easily understood within the confines of an individual healthcare organization, ideally the data are semantically interoperable (ie, have the same clinical and computational meaning) across organizations. Once standards have been implemented, the ability to capture complete data sets characterizes the data completeness domain. The ability to represent data via CDMs, to validate the accuracy of data, and to then use the data through automation of queries are additional domains that describe business capabilities related to data quality (see table 1).

Table 1

NESTcc data quality domain

NEST Stage	NESTcc data quality domain
	Consistency*	Completeness†	CDM‡	Accuracy§	Automation¶
1. Conceptual
2. Reactive	+	+	±
3. Structured	+	+	+	±
4. Complete	+	+	+	+	+
5. Advanced	+	+	+	+	+

*Data consistency: relevant uniformity of data across all hospitals, providers, and outpatients (eg, population/cohort identification, clinical documentation practices/policies between entities, workflow descriptions).

†Data completeness: presence of the necessary data elements for outcome assessment, use of common data elements, all data are electronically available and either complete or with little missing data.

‡Data models: CDMs include all data needed for decision making (eg, clinical data elements, unique device identifier).

§Data accuracy: EHR data are validated systematically, with comparison to the source, independent measurement, upstream data source, and known standard or valid values (eg, audits from charts).

¶Data automation: queries able to be run automatically against CDMs.

CDM, common data model; EHR, electronic health record; NEST, National Evaluation System for health Technology.

The Data Quality Maturity Model proposes the following five stages of increasingly advanced and integrated levels of performance for healthcare systems with respect to data management.

Stage 1—Conceptual: clinical processes capture data primarily in verbose documents, not as data; lack of organizational awareness of data utility, no effort to systematically manage healthcare data, lack of consistent or centralized governance, policies, and/or resources, data not organized centrally; data not available for organizational use and analysis; individual data units are project oriented or focused on immediate profits.

Stage 2—Reactive: the enterprise is able to react to requests for analysis and respond to research requests but mostly accomplished by manual chart review and abstraction; data management inefficient and expensive, with only sporadic recognition of data utility beyond immediate use; tacit support from leadership regarding need for centralized data governance and management, but only limited allocation of resources; data not available for organizational use and analysis beyond individual requests; individual data units are project oriented or focused on immediate profits.

Stage 3—Structured: clinical systems manage transactional data types (eg, orders, transactions, laboratory results, medication prescriptions) as discrete data; support from leadership (with resources provided) for centralized data governance and management of these data types at the enterprise level (eg, support for data exchange among internal systems); commitment to centralized enterprise data governance, management, and curation via managed processes, people, and technologies; non-administrative queries (clinical questions, research) conducted mostly as one-offs via individual queries, still moderate-to-high cost to extract data for analysis; able to support a CDM but not done routinely and automatically; data transmission to registries still largely accomplished by manual chart review and abstraction.

Stage 4—Complete: granular and complete clinical data based on standardized clinical common data elements captured in the processes of care, integrated into those care processes; UDI captured in the processes of care and available in EHR; health systems data routinely and systematically represent data externally via various CDMs, including efficient queries, support for large number of research projects; leadership provides centralized data governance, management, and curation at the enterprise level, ensuring performance and data quality of local units and achieving financial sustainability.

Stage 5—Advanced: data linkage and aggregation across systems enabled and open to external queries; interoperability of clinical data enabled; multiple sources of sustainable funding support for research; engagement of regulatory and industry enterprises with enterprise data; leadership responsible for centralized data governance, management, and curation at the enterprise level, business benefit well understood, with financial sustainability, and recognition and participation in initiatives external to the organization.

Conclusion

High quality data are critical to support the evaluation of medical devices and to inform regulatory and clinical decision making. The paper by Drozda et al showcases development of an innovative data source based on EHRs. The investment by healthcare systems into the standardization, curation and use of their RWD continues to grow rapidly but the success of these data sources is highly dependent on the quality of the data. The maturity models can help start an international dialog to drive consistency and a common language across healthcare enterprises that ultimately improves the quality of care.