Building and maintaining trust in clinical decision support: Recommendations from the Patient-Centered CDS Learning Network.

Knowledge artifacts in digital repositories for clinical decision support (CDS) can promote the use of CDS in clinical practice. However, stakeholders will benefit from knowing which they can trust before adopting artifacts from knowledge repositories. We discuss our investigation into trust for knowledge artifacts and repositories by the Patient-Centered CDS Learning Network's Trust Framework Working Group (TFWG). The TFWG identified 12 actors (eg, vendors, clinicians, and policy makers) within a CDS ecosystem who each may play a meaningful role in prioritizing, authoring, implementing, or evaluating CDS and developed 33 recommendations distributed across nine "trust attributes." The trust attributes and recommendations represent a range of considerations such as the "Competency" of knowledge artifact engineers and the "Organizational Capacity" of institutions that develop and implement CDS. The TFWG findings highlight an initial effort to make trust explicit and embedded within CDS knowledge artifacts and repositories and thus more broadly accepted and used.

INTRODUCTION

Clinical decision support (CDS) has been defined as a “process for enhancing health‐related decisions”1 that provides “clinicians, staff, patients, or other individuals with knowledge and person‐specific information, intelligently filtered or presented at appropriate times, to enhance health and health care.”2 CDS has become more available via Meaningful Use‐certified electronic health records (EHRs) and has been identified as a key component for disseminating clinical guidelines into clinical practice and achieving continuous improvement within Learning Health Systems.3, 4, 5

Despite its increasing availability, CDS arguably has not achieved its full value potential for impacting the costs, quality, or outcomes of care.6, 7 Significant limitations still exist for how evidence gets incorporated into routine clinical practice. Limitations include costs for developing CDS and non‐scalable implementations within and across health systems.8 To address these challenges, policy makers, developers, and researchers are exploring methods for encapsulating the clinical logic embedded in care guidelines into computable objects called “knowledge artifacts”9 and then offering those knowledge artifacts via publicly available repositories.10, 11, 12 A knowledge artifact represents evidence in machine readable code that invokes various actions via EHRs or other applications in clinical workflows such as patient‐specific alerts or documentation templates and order sets for providers. Those actions can be executed based on rules‐based logic or increasingly sophisticated algorithms. A knowledge artifact repository, like the Agency for Healthcare Research and Quality's CDS Connect (http://cds.ahrq.gov),13 is analogous to an “app store” wherein a customer can compare and contrast different tools to be used on a smartphone.

Thus, a repository makes CDS knowledge artifacts available to CDS developers and implementers for embedding within CDS tools and services. This approach holds promise for making CDS development more efficient and increasing the availability of shareable knowledge artifacts for CDS to care delivery organizations and ultimately their providers and patients within a CDS knowledge management life cycle. The Analytic Framework for Action (AFA) illustrates interconnected areas within a life cycle, which are (1) prioritizing clinical evidence to be transformed into an artifact, (2) authoring an artifact in ways that can be machine readable within an EHR, (3) implementing an artifact, (4) measuring an artifact's effects on care delivery and patient outcomes, and (5) contextual factors such as governance and legal requirements that influence how an artifact is managed and maintained over time (see Figure 1). These areas within the AFA ideally contribute to developing Learning Health Systems.14

Figure 1

The analytic framework for action depicts a CDS knowledge management life cycle

The success of CDS Connect and other knowledge artifact repositories will require not only further technical sophistication but also concomitant policies and governance procedures that help end users decide that they can trust knowledge artifacts prior to use. For example, if a community hospital wants to download a publicly available knowledge artifact for opioid prescribing CDS, how would it know in advance that the knowledge artifact is based on reliable evidence, that the artifact's evidence is routinely updated, and that third parties (eg, public or private payers and The Joint Commission) approve of the knowledge artifact and its evidence? Our work is premised on experience that trustworthy knowledge artifacts, and biomedical knowledge more broadly, will catalyze the development, distribution, measurement, and use of CDS for patient‐centered care within Learning Health Systems.4, 11 In this paper, we describe the efforts of the Patient‐Centered Clinical Decision Support Learning Network Trust Framework Working Group to (a) describe the people (“actors”) in the CDS ecosystem, and (b) consider their roles with respect to trust (eg, who needs to trust whom, and what they would need to know or demonstrate to ensure trust?). The purpose of this effort was to identify actionable recommendations that would promote trustworthiness of knowledge artifacts.

The Patient‐Centered Clinical Decision Support Learning Network (Learning Network) is funded by the Agency for Healthcare Research and Quality (U18 HS024849) to promote the dissemination of patient‐centered outcomes research into clinical workflows via CDS. The Learning Network chartered a Trust Framework Work Group (TFWG) to investigate ways that the CDS marketplace and research communities can establish and promote trust in knowledge artifacts and repositories. Toward that end, the TFWG had two goals: (1) identify barriers and facilitators to operationalizing a trust framework for one or more use cases, and (2) recommend how trust could promote fair, equitable, transparent, and trustworthy sharing of knowledge artifacts within a multi‐stakeholder CDS ecosystem. We provide the results that include attributes for trust, recommendations, and next steps that the field can take to promote trust in knowledge artifacts and repositories for CDS.

Trust and complex systems

Trust is a challenging multidimensional concept defined as one party's implicit “willingness to be vulnerable to another for a given set of tasks.”15 Trust in health systems is frequently evaluated in a number of approaches including terms of perceived fairness, fidelity to patients' best interests, system trust or confidence in policies and procedures, and confidentiality and privacy.12, 15, 16 Research examining the role of trust in interpersonal relationships frequently considers the honesty, competency, communication, or confidence in the reliability of relevant parties.12

Trust is a critical component of complex technical systems17 and is broadly recognized as a necessary attribute of health IT.18 Examples include those from a 2014 National Science Foundation workshop that identified trust as one of four broad system‐level requirements for a high functioning Learning Health System19 and the Office of the National Coordinator's draft Trusted Exchange Framework and Common Agreement (TEFCA).20 Yet, whereas TEFCA focuses on the trusted exchange of data, we pursued an investigation to make recommendations for the trusted exchange of knowledge.

The trust framework work group (TFWG)

We gathered 15 volunteer members from diverse backgrounds including clinicians, policy makers, and CDS vendors (see Acknowledgements) who met biweekly between February and August 2018. Members participated in moderated discussions, internal surveys, individual exercises, and iterative group editing of draft documents. A key exercise included members documenting aspects of trust from their respective perspectives that our group then distilled into fundamental attributes of trust (which we labeled as “trust attributes”) and recommendations. As this effort was exploratory and to our knowledge lacked a theoretical framework to build from, we iteratively vetted our efforts and the results with external stakeholders including participants in the CDS Connect Work Group. A more detailed description of the methods is detailed in the TFWG's white paper that is available online.21

DETERMINING CDS ACTORS, TRUST ATTRIBUTES, AND RECOMMENDATIONS FOR PROMOTING TRUST

The TFWG identified and agreed on definitions of actors, people within a CDS ecosystem that play one or more meaningful roles in prioritizing, authoring, implementing, and evaluating knowledge artifacts. The actors included patients, those within care delivery organizations (eg, clinicians and population health end users), vendors (eg, health IT vendors and Knowledge Distributors), payers, and more. We list the actors alphabetically in Table 1.

Table 1

Definition of actors participating in an ecosystem of knowledge translation and specification for implementation as CDS

Actors	Description	Examples
Clinicians	Medical professionals who care for patients.	Physicians, Nurses
Health IT Vendors	Commercial entities that provide health‐related technology solutions.	EHR vendors, CDS vendors, Health app developers
Knowledge Authors	Professionals such as domain experts and professional societies who write guidelines or other materials that provide clinical evidence to users in unstructured format (narrative text, image files, etc).*	United States Preventive Services Task Force, American College of Physicians
Knowledge Curators	Professionals who maintain knowledge artifact libraries and help ensure evidence is trustworthy (accurate, reliable, timely, etc).	Librarians, Knowledge Repository Analysts
Knowledge Distributors	Professional organizations that package, market, or sell knowledge artifacts as private organizations or in public‐private partnerships.	CDS Connect, First Databank
Knowledge Engineers	Professionals who translate clinical guidelines into artifacts in semi‐structured human readable form (L2)*, a computer interpretable form (L3)*, and/or machine‐executable formats (L4).*	Medical informaticists, Developers with clinical backgrounds
Governance Bodies	A governance body that reviews and approves CDS to be used in an organization or across networks.	Hospital CDS committees, Integrated health network knowledge management committees
Patients	Persons who are the ultimate decision‐makers in their healthcare and managing their health.	Adults, Guardians
Payers	Organizations that pay clinicians or patients for health‐related activities.	Blue Cross Blue Shield, United Healthcare
Policy makers	Persons who develop legal, regulatory, or policy guidance that guide care or payment.	Centers for Medicare and Medicaid Services, Food and Drug Administration
Population Health Analysts	Professionals who support clinicians, clinical teams, and patients by monitoring population health trends and recommending actions.	Care Managers, Care Coordinators, Public health professionals
Quality Improvement Analysts	Professionals who measure the impact of implemented CDS within health IT.	Researchers, Organization‐specific quality improvement specialists

L1‐L4 are Boxwala et al's four levels of knowledge abstraction interpretability from human readable (L1) to machine executable (L4).22

Taking into consideration the actors in the CDS ecosystem, their roles, and their responsibilities to one another, we developed and defined nine trust attributes, which provide different levels of consideration. For example, the “Competency” trust attribute represents the needs and expectations of an individual actor (eg, knowledge engineer) whereas “Organizational Capacity” represents organization‐level needs and expectations. Based on the nine trust attributes, the TFWG articulated 33 recommendations for action to ensure the attributes were reflected across the ecosystem (see Table 2).

Table 2

Trust attributes and their descriptions along with recommendations

Trust Attribute	Description	Recommendation
Competency	An actor who authors a knowledge artifact is deemed to be competent in the role played in the CDS ecosystem. For example, an author of a knowledge artifact should be judged competent, qualified, and an appropriate authority to develop the artifact based on factors such as past performance, professional qualifications, or certifications.	1.1 Authors have descriptions with background information including affiliations, years participating, and frequency of participation. 1.2 Authors promote respect and dignity when providing feedback. 1.3 Authors are credentialed by an agreed‐upon entity through education or training, experience, and dependability. 1.4 Knowledge professionals are certified that they are competent in the knowledge management life cycle; (Wright et al., 2009, pp 334‐346; Glaser and Hongsermeier, 2014) can competently interpret, encode, and execute knowledge; and are competent in addressing issues of conflict of interest. 1.5 Competency should apply to both individuals and organizations.
Compliance	A knowledge artifact should conform to defined standards and criteria including copyright and intellectual property.	2.1 Knowledge artifacts provide human‐readable and machine‐readable forms (whenever applicable) as well as supporting references. 2.2 Knowledge artifacts are implemented in compliance with best practices for safe and effective implementation. 2.3 Knowledge artifacts are encoded using current standards for controlled medical terminologies, value sets, clinical data models, and knowledge representation formalisms.
Consistency	A knowledge artifact should repeatedly generate expected results over time when given requisite inputs (eg, patient data or supporting CDS triggers).	3.1 Authors take on responsibility of ensuring accurate knowledge translation and specification of a knowledge artifact.
Discover‐ability & Accessibility	The evidence behind an executable knowledge artifact is documented (discoverable) from metadata associated with the artifact. Artifacts and their contents have clear and appropriate reasoning for recommendations available to the end users. Artifacts are accessible to potential users, including patients and policy makers.	4.1 Knowledge is made accessible through search technology in conjunction with effective and helpful key terms. 4.2 Knowledge can be reliably searched for and found over time, so that users can find the same knowledge across successive versions. 4.3 References to supporting evidence are clearly labeled and linked (preferably deep linked) to relevant supporting information. 4.4 Data that inform an artifact can be found and accessed.
Evidence‐based	The evidence instantiated within an artifact must apply to the clinical condition it is meant to support. Limitations are stated clearly, and the evidence supporting the clinical guideline/predictive model, etc, in an artifact is substantiated and has clear clinical appropriateness.	5.1 Metadata indicate the date that evidence was originally published and the date that evidence was last reviewed. 5.2 Metadata state any known limitations, restrictions, or exclusions to any given evidence. 5.3 Artifacts contain references to the evidence base on which they are built, including both narrative guidelines and the data supporting those guidelines. 5.4 Artifacts include metadata for all supporting citations. 5.5 Artifacts include evidence about their methods (eg, order set v. alert), usage history, and available outcomes.
Feedback and Updating	Stakeholders have the functional ability to provide timely feedback and suggest improvements to a knowledge artifact. Feedback may be directed to diverse actors in the ecosystem (knowledge engineers, knowledge authors, etc).	6.1 Systems capture error logs and feedback about an artifact within the context of its use (e.g., EHR system, clinical setting, crash data etc.). 6.2 Systems provide feedback mechanisms including means for users to ask questions about an artifact's context of use. 6.3 Metadata capture the dates an artifact was first and last published, with update dates in between. 6.4 Artifacts contain auditable records of updates and changes over time. 6.5 Artifacts are updated based in part on feedback from their operational performance over time. 6.6 Authors provide bidirectional feedback to one another to rate (and improve) each other's work.
Organizational Capacity	An organization that sponsors knowledge artifact development or implementation (or both) should have the necessary funding, staffing, and resources to maintain a knowledge artifact and measure its effect(s).	7.1 Develop skills and capacity of staff, systems, and resources that support implementation, ongoing evaluation, feedback, communications, and governance. Include implementation guidance with artifacts that conveys the necessary resources to implement that artifact. 7.2 Knowledge artifacts include implementation guidance that conveys the necessary resources to implement that artifact.
Patient‐centeredness	When possible, a knowledge artifact should leverage patient‐centered outcome research findings and/or patient‐specific information (the patient's clinical data, patient preferences, patient‐generated health data, patient‐reported outcomes) to support decisions by individual patients, their approved caregivers, and/or their care teams.	8.1 Requirements for patient‐level or patient‐generated data input are clearly indicated. 8.2 Evidence that accounts for patient‐level or patient‐generated data is clearly indicated. 8.3 Consent for use of patient‐level or patient‐generated data is clearly indicated.
Transparency	A knowledge artifact should be applied and used ethically to clearly convey all potential conflicts of interest and disclosures of interest related to its development or recommendation to detect bias or discrimination in its use.	9.1 Clearly indicated policies describe the procedures for implementing, updating, revising, and removing artifacts. 9.2 Clearly indicated policies address conflict of interest. 9.3 Knowledge artifacts are consistently implemented with licensing agreements and any secondary use rights are explicit. 9.4 Knowledge artifacts are consistently implemented in ways that support equity in health and health care.

DISCUSSION

The TFWG examined the issue of defining, building, and maintaining trust among actors who develop, exchange, implement, or use knowledge artifacts for CDS. We identified 12 relevant actors (see Table 1) and developed nine trust attributes with 33 associated recommendations (see Table 2). These findings represent to our knowledge the first time the elements of trust for knowledge artifacts within a CDS ecosystem have been comprehensively defined. We address the trust attributes within four knowledge management life cycle domains below as depicted within the AFA (see Figure 1): prioritizing, authoring, implementing, and measuring impact. Each trust attribute is identified in italics.

Trust attributes for prioritizing evidence: Evidence‐based and patient‐centeredness

Accurate and reliable evidence is essential for trust in any knowledge artifact used in a CDS system. Repositories and the artifacts within should integrate a formal Evidence‐based rating system such as GRADE23 so that end users can assess and weigh the quality of the evidence within a knowledge artifact for CDS. In addition, the evidence should be interpreted and applied in a Patient‐centered manner whenever possible given a decision context and includes unique patient data and context, patient preferences, patient‐reported outcomes, or other patient‐generated data.

Trust attributes for authoring: Competency, consistency, and discovery and accessibility

Authoring‐related trust attributes include considerations around the Competency (qualifications and past performance) of artifact authors, as well how well and reliably they implement knowledge artifacts that lead to consistency in the use of CDS. Competency could be assessed by the community or by a governing body such as a professional society certification, vendor certifications, or licensure boards as well as by authors' experience and previous track records. Consistency relates to the reliable and consistent performance of an implemented knowledge artifact as CDS across disparate implementations of health IT as well as across different care delivery systems or settings of care. CDS Hooks represents one emerging standard and solution for consistently and reliably triggering the logic within knowledge artifacts. Discoverability and Accessibility extends to the evidence trail and/or the provenance of a knowledge artifact and should be traceable to the sources such as clinical guidelines.

Trust attributes for implementation: Organizational capacity, compliance, and transparency

This is essential to be both Compliant with the current best practices for knowledge representation standards and achieving “the 5 rights” for CDS implementation.1 Implementing organizations (eg care delivery organizations, IT vendors, and knowledge vendors) must have the Organizational Capacity to safely and effectively implement CDS, monitor its use, and keep the implemented CDS up to date. This suggests that an organization's EHR and data readiness for implementing knowledge artifacts are directly linked to the quality of expected outcomes produced by the artifact and its trustworthiness in practice. Maturity models for EHR and health IT infrastructures may be useful in assessing initial capacity for knowledge artifact implementations but could also be further developed and extended to consider Organizational Capacity for adopting use of knowledge artifacts. For example, the United States Food and Drug Administration is considering an organization‐level approach as part of its Software as a Medical Device (SaMD) pre‐certification process..24 Finally, full Transparency must exist in the implementation to capture any assumptions made, deviations from guideline evidence logic, or other changes in data structures used in CDS. We refer readers to the work of the Center for Open Science's Transparency and Openness Promotion guidelines for considerations in this area.25, 26

Trust attribute for measurement: Feedback and updating

Key to Learning Health Systems is the capacity to provide Feedback and Updates on the implemented knowledge artifact or CDS from the vantage point of any user: whether that be a physician, nurse, or other member of the care team, as well as the patient him or herself.3 Feedback and Updates may include an end user's subjective assessment, as well as more quantitative assessments of impact. These may include the methods for measuring CDS impact on near‐ and long‐term process‐level and patient‐level outcomes. Feedback ought to occur at multiple levels: from a user to the system implementers, to the CDS author, IT system designers, and potentially even to the creators of the primary evidence.

The areas we outlined above have significant implications for promoting trust in the ways clinical knowledge is built and maintained such as noting common metadata schema across public and private knowledge repositories, a direct linkage to primary source documentation, and any ability to determine that the evidence applies in an appropriate manner to the patient context at hand.

FUTURE WORK

CDS Connect is applying the trust attributes and recommendations for promoting trust to the development of its platform and metadata schema. Future efforts should focus on linking trust attributes to policy, governance, and translation into practice. For example, we foresee further explication of the Competency trust attribute and providing recommendations as to how Competency can be economically operationalized to help prospective end users inspect and compare offerings. Policies for ensuring the validity of CDS encoded in knowledge artifacts or standardized labeling for knowledge artifacts would help systems such as CDS Connect become scalable enterprises but require further research to ensure policies and standards are evidence‐based.

Gaps identified in the development of the trust framework also point to areas where future capabilities might be developed. We are excited about the prospect of reporting systems that enable the Feedback and Updating trust attribute such as the automatic submission of CDS and EHR performance data for knowledge artifacts. We believe that attribute would also be of great value to key actors (authors, implementers, policy makers) and would be a major step toward supporting compliance in Learning Health Systems. We also believe that an important area of future work will be designing for Patient‐Centeredness in repositories, such as providing robust means for patients themselves to compare and contrast artifacts for personal use or use of metadata that inform potential users in the ways that evidence is patient‐centered. In parallel, we believe additional work to promote Transparency in patented knowledge will better guide stakeholders how to develop and implement knowledge artifacts (or not).

We anticipate future work in trust for CDS knowledge artifacts will refine the trust attributes themselves, and the recommendations, based on real‐world experience. An area for further investigation would be whether and how levels of trust vary by actors; for example, the degrees to which providers versus patients trust—or perceive the need to trust—CDS knowledge artifacts. We expect further work will also explore potential trust attributes related to knowledge artifact security (eg, intellectual property and provenance), the issues of which differ from data security that TEFCA addresses. We furthermore hope to develop methods (assessment instruments or rating scales) that may be based upon the attributes to develop one or more trust metrics for knowledge artifacts. In this area, we are tracking the exciting developments coming out of the HL7 CDS Work Group that include EHR standards for interoperable clinical guidelines—CPGonFHIR27—and interoperable systematic reviews—EBMonFHIR.28 Each of these efforts seeks to enable streamlined exchanges of knowledge through standardized and computable artifacts for CDS and if successful could scale clinical knowledge exchange beyond current capabilities. However, more efficient exchange is unlikely to promote use (and reuse) of that knowledge unless care delivery organizations, providers, and patients can trust artifacts' accuracy and timeliness. We are contributing to these efforts to inform stakeholders how trust plays a foundational role critical to collective success, and we are thankful for the input and openness of CDS Connect developers who have been considering our work to inform their design decisions.

CONCLUSION

Shareable and computable knowledge artifacts for CDS have long been a goal within informatics given the potential to more effectively integrate biomedical knowledge into EHRs and Learning Health Systems. Trust in knowledge artifacts will be a key feature of promoting and sustaining a knowledge‐sharing ecosystem composed of multiple stakeholders and information systems. We identified actors in a CDS ecosystem, trust attributes, and recommendations that can enhance knowledge artifacts that support efforts for their adoption and implementation. We advocate for further efforts in this area to advance the trustability of biomedical knowledge and promote its implementability and scalability to make CDS effective in Learning Health Systems.

CONFLICT OF INTEREST

Dr Middleton receives salary as an employee and stock holder of Apervita, Inc, receives funding from the Gordon and Betty Moore Foundation research support, and is a consultant to the AHRQ‐funded Patient‐centered CDS Learning Network. All other authors report no potential conflicts of interest.

FUNDING INFORMATION

We would like to acknowledge the funding from Agency for Healthcare Research and Quality Cooperative Agreement (U18 HS024849) that made the TFWG's work possible. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.