A scoping review of knowledge authoring tools used for developing computerized clinical decision support systems.

OBJECTIVE: Clinical Knowledge Authoring Tools (CKATs) are integral to the computerized Clinical Decision Support (CDS) development life cycle. CKATs enable authors to generate accurate, complete, and reliable digital knowledge artifacts in a relatively efficient and affordable manner. This scoping review aims to compare knowledge authoring tools and derive the common features of CKATs.
MATERIALS AND METHODS: We performed a keyword-based literature search, followed by a snowball search, to identify peer-reviewed publications describing the development or use of CKATs. We used PubMed and Embase search engines to perform the initial search (n = 1579). After removing duplicate articles, nonrelevant manuscripts, and not peer-reviewed publication, we identified 47 eligible studies describing 33 unique CKATs. The reviewed CKATs were further assessed, and salient characteristics were extracted and grouped as common CKAT features.
RESULTS: Among the identified CKATs, 55% use an open source platform, 70% provide an application programming interface for CDS system integration, and 79% provide features to validate/test the knowledge. The majority of the reviewed CKATs describe the flow of information, offer a graphical user interface for knowledge authors, and provide intellisense coding features (94%, 97%, and 97%, respectively). The composed list of criteria for CKAT included topics such as simulating the clinical setting, validating the knowledge, standardized clinical models and vocabulary, and domain independence. None of the reviewed CKATs met all common criteria.
CONCLUSION: Our scoping review highlights the key specifications for a CKAT. The CKAT specification proposed in this review can guide CDS authors in developing more targeted CKATs.

INTRODUCTION

Clinical Decision Support (CDS) is a key component of healthcare transformation to achieve the Quadruple Aim. Computerized clinical decision support systems enable the wide and fast adoption of CDS among health systems. Once integrated with the clinical workflow, CDS systems provide users with targeted information that is intelligently filtered to assist clinicians at the point of care. To achieve a successful CDS system in clinical practice, several factors need to be in place such as having the right information represented in the CDS, CDS producing the right intervention format, using CDS through the right channels, deploying the CDS in the right clinical workflow, and the CDS system being used by the right person.,

Development and maintenance of CDS knowledge, also known as Computable Biomedical Knowledge (CBK), is a major challenge in healthcare. Dissemination of CBKs, which are almost always published as narrative text and diagrams, requires transformation into a computable format, yet such transformation is a tedious process taking time and resources to ensure currency.

Clinical Knowledge Authoring Tools (CKATs) are used to generate accurate, complete, and reliable digital knowledge artifacts in a relatively efficient and affordable manner. Fast and efficient CKATs are increasingly needed for CBK development since: (1) CDS systems are quickly becoming an essential tool for healthcare providers; (2) EHRs are ubiquitously used in most inpatient and outpatient settings in the United States; (3) regulators and clinical quality officers have established metrics motivating health care institutions to use CDS; and (4) practitioners, being members of the Internet generation, expect their computer-based tools to provide decision support. CKATs range from simple text editors to complex software solutions such as the Arden Syntax editor.

Continuously assessing and updating CBK is crucial to make the CDS process effective and timely. However, the volume of available clinical evidence is increasing at a rapid pace, thus requiring tools, such as CKAT, to frequently update and adjust the CBK. To address this challenge, CKATs are increasingly connecting the Knowledge Engineering and Knowledge Use components of a CDS systems. Hence, CKATs can reduce the overall cost of CKB development by (1) taking the anticipated clinical workflow into account and (2) continuously improve and deploy CBK models into clinical settings (Figure  1).

Figure 1.

Knowledge authoring tool transforming the knowledge into an actionable clinical decision support format and continuously improving its quality and performance.

A comprehensive CKAT is responsible for authoring, reviewing, testing, certifying, publishing, and assessing CDS models. Several different types of users collaborate in the process of a CBK model development life cycle driven by the CKAT, such as subject matter experts, clinical experts, developers, data scientists, clinical champions, and administrators.

CKAT systems have increasingly incorporated knowledge extraction mechanisms in addition to knowledge authoring tools. In such a hybrid approach, CKATs not only enable the knowledge curators to translate CPGs and other medical evidence into CKB but also enable end-users to generate de-novo knowledge from a clinical data repository (eg, generating a statistical model that can be integrated into a CDS system). Thus, knowledge curators are gradually incorporating statistical and machine learning tools (eg, Orange, RapidMiner, Weka, KNIME) in parallel with the CKAT systems. These model-authoring tools help knowledge engineers to extract, validate, and author CBK at once. Since statistical and machine learning tools are not primary CKATs, those tools are not included in this scoping review.

Even though extensive review studies have been conducted on CDS systems (eg, types and effectiveness of such systems), research is lacking on the types and specifications of CKATs. Given the variety and variability across CKATs, this study aims to address the following questions: (1) what are the widely published CKATs? and (2) what are the salient features of those CKATs?

MATERIALS AND METHODS

Eligibility criteria

Our criteria for inclusion of reviewed papers are as follows: (1) Quantitative and qualitative articles that focused on clinical knowledge authoring. (2) Studies on the use of ontology and standard models as part of CDS authoring. (3) Studies published in peer-reviewed journals or conference proceedings (ie, editorials, commentaries, letters, reviews, and opinion articles were excluded). (4) Articles published in English. (5) Published after 2000, as our screening query found few publications mentioning computerized CKATs prior to 2000.

Information sources and search strategy

Search strategies were constructed to identify (1) peer-reviewed, published literature addressing the role of rule authoring environment, and (2) additional snowball searches to identify prominent tools currently used in the CDS systems.

Our primary literature search used PubMed and Embase search engines. PubMed was searched for relevant articles using the keyword “Clinical Decision Support Knowledge Authoring,” which resulted in 1467 records. This initial search strategy was then developed iteratively for the PubMed database, and once all authors were satisfied with both the breadth and specificity of the results, this strategy was translated for the other databases. The final PubMed search strategy, conducted on the legacy PubMed interface, included the following combination of key terms: (“decision support systems, clinical” [MeSH Terms] OR “clinical decision support” [All Fields] OR “clinical guideline*” [All Fields]) AND “author*” [All Fields] AND Data range: Publication date from 2000/01/01.

This report fulfills the PRISMA checklist items for scoping reviews. Search results were downloaded into a reference management software to facilitate the removal of duplicate citations, and the resulting unique set of citations underwent title/abstract and full-text screening. During the review process, features articulated by the manuscripts were abstracted and encoded in spreadsheets. Shortlisting the features was accomplished by applying the thematic analysis approach to the abstracted data.

RESULTS

Our search strategy returned 1579 publications, 1494 from PubMed and 85 from EMBASE. We removed 9 duplicates, 1157 nonrelevant abstracts, and 366 articles lacking CKAT details. We included 47 articles in our final review (Figure  2). These articles included 33 unique CKATs. We used the reviewed papers to compose the list of criteria for CKAT, which included topics such as simulating the clinical setting, validation/testing details, compliance, transparency, intellisense (ie, usability features for coding), standard clinical models and vocabulary, and domain independence.

Figure 2.

Article-flow diagram based on the PRISMA guideline.

The final list of articles and CKATs included in those studies were populated. Several articles used the same CKAT; however, only a few papers analyzed more than 1 CKAT (Table  1).

Table 1.

Articles included in the review and CKATs mentioned in each study

No.	Author	Year	Article title	CKAT
1	Kerexeta et al	2020	Adaptative clinical decision support system using machine learning and authoring tools24	KGT (EXCON)
2	Richardson et al	2020	Building and maintaining trust in clinical decision support: Recommendations from the Patient-Centered CDS Learning Network25	CDS Connect
3	Torres et al	2020	A domain-independent semantically validated authoring tool for formalizing clinical practice guidelines26	Authoring Tool
4	Lomotan et al	2020	To share is human! Advancing evidence into practice through a national repository of interoperable clinical decision support27	CDS Connect
5	Heen et al	2020	A framework for practical issues was developed to inform shared decision-making tools and clinical guidelines28	MAGICapp
6	Fox et al	2020	OpenClinical.net: Artificial intelligence and knowledge engineering at the point of care29	OpenClinical
7	Totten et al	2019	Improving access to and usability of systematic review data for health systems guidelines development30	MagicApp
8	Zhang et al	2018	Using systematic reviews in guideline development: The GRADE approach31	GrADEpro
9	Choi et al	2018	Artificial intelligence clinical decision supporting system for diagnosis of heart failure: Concordance with expert decision32	I-KAT
10	Piovesan et al	2018	GLARE-SSCPM: an intelligent system to support the treatment of comorbid patients33	GLARE
11	Alkasab et al	2017	Creation of an open framework for point-of-care computer-assisted reporting and decision support tools for radiologists34	Marval
12	Ali et al	2017	Multi-model-based interactive authoring environment for creating shareable medical knowledge35	I-KAT
13	Zini et al	2017	An environment for guideline-based decision support systems for outpatients monitoring36	Alium
14	Zhang et al	2016	A concise drug alerting rule set for Chinese hospitals and its application in computerized physician order entry37	Drug alerting rule authoring tool
15	Lin et al	2015	Design, development, and initial evaluation of a terminology for clinical decision support and electronic clinical quality measurement38	OpenCDS
16	Khodambashi et al	2015	Filling the gap between guideline development and formalization process—a requirement analysis39	GRADEpro MAGICapp
17	Zhang et al	2015	Mobilizing clinical decision support to facilitate knowledge translation: a case study in China40	Knowledge authoring web portal
18	Kristiansen et al	2015	Development of a novel, multilayered presentation format for clinical practice guidelines41	MagicApp
19	Ali et al	2014	Arden syntax studio: Creating medical logic module as shareable knowledge10	Arden syntax studio I-KAT
20	Ali et al	2014	Customized clinical domain ontology extraction for knowledge authoring tool42	I-KAT
21	Sottara et al	2014	The health eDecisions authoring environment for shareable clinical decision support artifacts43	HeD Editor
22	Ali et al	2013	Authoring tool: acquiring sharable knowledge for Smart CDSS44	Smart CDSS Authoring tool
23	Kim et al	2013	Design of shareable and interoperable clinical decision support system architecture45	SAGE Authoring Environment
24	Pasche et al	2013	Assisted knowledge discovery for the maintenance of clinical guidelines46	KART
25	Colantonio et al	2012	A knowledge editing service for multisource data management in remote health monitoring47	Knowledge Editing Service (KES)
26	Shiffman et al	2012	Building better guidelines with BRIDGE-Wiz: Development and evaluation of a software assistant to promote clarity, transparency, and implementability48	BRIDGE-Wiz
27	Kim et al	2011	Implementation of guideline-based CDSS49	SAGE Authoring Environment
28	Song et al	2011	A multi-classifier based guideline sentence classification system50	Clinical process modeling toolkit
29	Pasche et al	2011	KART, a knowledge authoring and refinement tool for clinical guidelines development51	KART
30	Kam et al	2011	Integration of heterogeneous clinical decision support systems and their knowledge sets: feasibility study with drug-drug interaction alerts52	SAGE
31	Cho et al	2010	Design and implementation of a standards-based interoperable clinical decision support architecture in the context of the Korean EHR53	SAGE Authoring Environment
32	Shiffman et al	2010	Writing clinical practice guidelines in controlled natural language54	ACE Authoring Tool
33	Höhne et al	2010	An internet portal for the development of clinical practice guidelines55	Internet Portal
34	Koch et al	2010	Representation of clinical nursing protocols using GEM II and GEM Cutter56	GEM Cutter
35	Regier et al	2009	A clinical rule editor in an electronic medical record setting: development, design, and implementation57	Rule Editor
36	Dunsmuir et al	2008	A knowledge authoring tool for clinical decision support58	SmartCare
37	Hussain et al	2008	An ontology-based framework for authoring and executing clinical practice guidelines for clinical decision support systems59	CPG-EX
38	Kim et al	2008	Knowledge translation of SAGE-based guidelines for executing with knowledge engine60	SAGE
39	Hussain et al	2007	Ontology driven CPG authoring and execution via a semantic Web framework61	CPG-EX
40	Hulse et al	2005	KAT: A flexible XML-based knowledge authoring environment62	KAT
41	Skonetzki et al	2004	HELEN, a modular framework for representing and implementing clinical practice guidelines63	HELEN Guideline Editor
42	Berg et al	2004	SAGEDesktop: An environment for testing clinical practice guidelines64	SAGEDesktop
43	Votruba et al	2004	Tracing the formalization steps of textual guidelines65	Guideline Markup Tool
44	Gennari et al	2003	The evolution of Protégé: An environment for knowledge-based systems development66	Protégé
45	Peleg et al	2002	Support for guideline development through error classification and constraint checking67	GLIF3 Authoring Tool
46	Clercq et al	2001	Design and implementation of a framework to support the development of clinical guidelines68	KA-Tool
47	Humber et al	2001	Medical decision support via the internet: PROforma and Solo69	PROforma

CKAT: Clinical Knowledge Authoring Tool.

Technical aspects of the CKATs were extracted and merged, if needed, from the identified articles (Table  2). To review the development platform characteristics, we analyzed the supported operating systems, the type of the application, and the programming languages used. Most of the CKATs are web-based applications requiring only a web browser, hence independent from the operating systems. Java and JavaScript are the major programming languages used. Among the reviewed CKATs, 55% are open source, letting others further expand on the existing knowledge authoring core. The majority of the CKATs are using, either directly or indirectly through a programming interface, medical terminology standards such as SNOMED-CT (Systematized Nomenclature of Medicine—Clinical Terms), LOINC (Logical Observation Identifiers Names and Codes), UMLS (Unified Medical Language System), ICD (International Classification of Diseases), and RxNORM (Medication Normalized Naming System). Of the reviewed CKATs, 70% support application programming interface (API) integration with other information system platforms such as EHRs (eg, CKAT authorizing the EHR systems to submit assessment values and then pulling different CDS scenarios synchronously). Among the analyzed CKATs, all but one (Rule Editor) support some version of a graphical user interface (GUI), which facilitates the knowledge authoring and review process by clinicians and informatics experts.

Table 2.

Development characteristics of reviewed CKATs

CKAT no.	CKAT	Paper no.	Developers and funders	Development platform		Open source	Medical terminology standards	API	Inform flow	GUI
				Type	Programming language
1	KGT (EXCON)	1	Basque Government's ELKARTEK 2017 program under EXCON project	Web based	R	No	No	Yes	Yes	Yes
2	ACE Authoring Tool	32	National Library of Medicine and the Agency for Healthcare Research and Quality (AHRQ)	Web based	JavaScript	Yes	UMLS SNOMED LOINC	No	Yes	Yes
3	Authoring Tool (TORRES et al.)	3	eHeatlh and Biomedical Applications, Vicomtech, Donostia-San Sebastian,	Desktop	Unk	No	Unk	Yes	Yes	Yes
4	Alium	13	developed by Deontics Ltd (London, UK)	Web based	JavaScript	No	ICD SNOMED	Yes	Yes	Yes
5	BRIDGE WIZ	26	National Library of Medicine and the Agency for Healthcare Research and Quality	Desktop	Java	No	No	Yes	Yes	Yes
6	CDS Connect	2, 4	AHRQ funded	Web based	Unk	Yes	UMLS ICD SNOMED	Yes	Yes	Yes
7	CPG-EX	37, 39	Dalhousie University	Web based	Java	Yes	UMLS	Yes	Yes	Yes
8	Knowledge -authoring web portal	14	DaYi hospital	Web based	Unk	No	ICD	No	Yes	Yes
9	GLIF3 Authoring Tool	45	National Library of Medicine and by the Telemedicine, U.S Army medical research	Web based	Unk	Yes	Unk	Yes	Yes	Yes
10	GrADEpro	8, 16	WHO, McMaster University	Web based	Unk	No	Unk	Yes	Unk	Yes
11	Guideline Markup Tool	43	Austrian Science Fund, European Commissions IST program	Web based	Java	No	Unk	No	Yes	Yes
12	HeD Editor	21	U.S. Office of the National Coordinator for Health Information Technology, SHRPC project 2B	Web based	JavaScript, AngularJS, d3js	Yes	SNOMED LOINC RxNORM	Yes	Yes	Yes
13	I KAT	9, 12, 19, 20	Ministry of Knowledge Economy, Korea, under the ITRC, National Library of Medicine	Web based	Java	Unk	SNOMED	Yes	Yes	Yes
14	KAT	40	HUG—Hospital of Geneva	Web based	Java	No	SNOMED	Yes	Yes	Yes
15	KART	24, 29	Unk	Unk	Unk	Yes	SNOMEDICD	Yes	Yes	Yes
16	KA-Tool	46	Unk	Unk	Unk	Unk	Unk	Yes	Yes	Yes
17	Knowledge Editing Service (KES)	25	European Union Information Society Technologies Project	Web based	Java	Yes	Unk	Yes	Unk	Yes
18	MAGICapp	5, 7, 18	MAGIC evidence ecosystem foundation	Web based	JavaScript	Yes	ICPC ICD SNOMED-CT ATC RxNORM MeSH	Yes	Yes	Yes
19	CAR/DS Authoring Tool	11	American College of Radiology	Web based	JavaScript	Yes	RadLexLOINCSNOMEDRADElements CDE	No	Yes	Yes
20	SmartCare	36	Canadian Institute of Health Research	Web based	Java	Yes	SNOMED	Yes	Yes	Yes
21	PROforma Authoring Tool	47	Royal Hospital London, NHS	Unk	Unk	Yes	Unk	No	Yes	Yes
22	OpenCDS	15	National human genome research institute and University of Utah Department of Biomedical Informatics	Web based	Java	Yes	SNOMED ICD RxNORM	Yes	Yes	Yes
23	OpenClinical	6	Cancer Research U.K. and later at Oxford University and UCL/Royal Free Hospital in London.	Web based	Unk	Yes	Unk	Unk	Yes	Yes
24	Rule Editor	35	Partners HealthCare	Web based	Unk	No	Unk	Unk	Yes	No
25	SAGE Authoring Environment	23, 27, 31	Unk	Web based	Unk	Unk	SNOMED	Unk	Yes	Yes
26	SAGEDesktop	42	SAGE project partners are: Apelon Inc., IDX Systems, Intermountain Health Care, Mayo Clinic—Rochester, Stanford Medical Informatics, and the University of Nebraska Medical Center	Web based	Unk	No	SNOMED LOINC	Yes	Yes	Yes
27	SAGE	30, 38	National Research Foundation of Korea	Web based	Java	No	Unk	Yes	Yes	Yes
28	Smart CDSS Authoring tool	22	Ministry of Knowledge Economy, Korea	Web based	NET, Java	Yes	SNOMED	Yes	Yes	Yes
29	Protégé	44	Knowledge Modeling Group at Stanford Medical Informatics	Web based	Java	Yes	Unk	Yes	Yes	Yes
30	Internet Portal	33	German Association of the Scientific Medical Associations	Web based	Python	Yes	Unk	No	Yes	Yes
31	GEM Cutter	34	Mayo Clinic, Yale School of Medicine	Web based	Unk	Yes	SNOMED LOINC RxNORM	No	Yes	Yes
32	GLARE	10	Universita del Piemonte Orientale	Web based	Java	No	SNOMED ATC	Yes	Yes	Yes
33	SmartCare	36	Canadian Institutes of Health Research	Web based	Java	Yes	SNOMED	Yes	Yes	Yes

Abbreviations: API: Application Programming Interface; ATC: Anatomical Therapeutic Chemical Classification System; CKAT: Clinical Knowledge Authoring Tool; GUI: Graphical User Interface; ICD: International Classification of Diseases; ICPC: International Classification of Primary Care; Indep: Independent; LOINC: Logical Observation Identifiers Names and Codes; MeSH: Medical Subject Headings; OS: Operating System; RADElements: Radiology Elements; RadLex: Radiology Lexicon; RxNORM: Medication Normalized Naming System; SNOMED: Systematized Nomenclature of Medicine; Unk: Unknown; UMLS: Unified Medical Language System.

Different authoring environment characteristics of the CKATs were extracted from the reviewed articles (Table  3). Of the 33 CKATs, 27 support at least 1 standard language for knowledge encoding. Some CKATs went through multiple revisions using different programming languages. Moreover, 36% of the CKATs have a built-in version control feature. Of the CKATs, 28 support CDS authoring independent of any domain/use case. Even though 97% of the CKATs support GUI, only 70% facilitates collaborative knowledge authoring; 67% of the CKATs support simulating the clinical setting to ensure CDS works as expected at the point of care. Only 18% of the CKATs support grading the evidence, while 79% of them support testing all possible scenarios. As part of the knowledge base updating cycle, CKATs assess the knowledge deployed by receiving feedback from the CDS system. Among the reviewed CKATs, 61% support this surveillance feature. And, 97% of CKATs support intellisense features to help the authors while encoding the knowledge (eg, automatically pulling the values from a terminology standard system, and color coding the scenarios not reachable); 48% of the CKATs support automated CDS content publishing, facilitating the content deployment to CDS systems, especially when multiple users collaborate in the CDS generation process (Figure  3).

Table 3.

Application specification of reviewed CKATs

CKAT no.	CKAT name	Knowledge authoring language	Version control	Domain and specialty	Collaboration authoring	Use cases	Simulate clinical setting	Grading evidence	Validation and testing	Surveillance and assessing	Intellisense	Continuous deployment
1	KGT (EXCON)	Unk	Unk	Domain Indep	Unk	Readmission of diabetic patients	No	Yes	Yes	No	Yes	No
2	ACE Authoring Tool	ACE, Arden Syntax, SWRL	No	Pediatric	No	Initial Urinary Tract Infection in Febrile Infants and Young Children	Yes	No	No	No	No	No
3	Authoring Tool (TORRES et.al)	Arden syntax, RDF, OWL	No	Domain Indep	Unk	Gestational diabetes, current physical activity level	Unk	Yes	No	No	Yes	Yes
4	Alium	PROforma	Yes	Domain Indep	Unk	Prevention, diagnosis and treatment of therapy side effects, head and neck cancer mobile devices support	Yes	Yes	Yes	Yes	Yes	Yes
5	BRIDGE WIZ	GEM GLIA	Yes	Domain Indep	Yes	Diabetes type II	Unk	Yes	Yes	Yes	Yes	Yes
6	CDS Connect	CQL	Yes	Domain Indep	Yes	CVD, chronic pain management	No	No	No	Yes	Yes	Yes
7	CPG-EX	Jena CPG syntax, OWL	No	Domain Indep	Unk	Radiology—according to the EU radiation protection 118	Yes	No	Yes	Yes	Yes	Yes
8	Knowledge-authoring web portal	Unk	No	Domain Indep	Unk	Pharmacy drug override, drug check use	No	No	Yes	No	Yes	Unk
9	GLIF3 Authoring Tool	GLIF3	Unk	Domain Indep	Unk	Migraine headache	Yes	No	No	No	Yes	Yes
10	GrADEpro	Unk	Unk	Domain Indep	Yes	WHO Interim policy guidance on the use of medication in the treatment of tuberculosis	No	Yes	Yes	Yes	Yes	Unk
11	Guideline Markup Tool	Asbru	No	Domain Indep	No	Unk	No	No	Unk	Yes	Yes	Unk
12	HeD Editor	HeD expression language, OWL2-DL	Unk	Domain Indep	Yes	Antithrombotic therapy on discharge adapted from MQF-00685 quality measure	Yes	Unk	Unk	Yes	Yes	Yes
13	I KAT	Arden syntax	Unk	Domain Indep	Yes	Head and neck cancer	Yes	Unk	Unk	Yes	Yes	Unk
14	KAT	Arden syntax	Unk	Domain Indep	Yes	POE	Yes	No	Yes	Unk	Yes	No
15	KART	RDF, Query Language (SPARQL)	Unk	Domain Indep	Yes	Antibiotic prescribing and many more	Yes	Unk	Yes	Unk	Yes	Unk
16	KA-Tool	GLIF3	Yes	Domain Indep	No	ICU based	No	No	Yes	Unk	Yes	Yes
17	Knowledge Editing Service (KES)	OWL	Unk	remote monitoring	Yes	COPD, diabetes	No	No	Yes	Yes	Yes	Yes
18	MAGICapp	Unk	Unk	Domain Indep	Yes	Various – 15 BMJ guidelines, COVID-19	Unk	Yes	Yes	Yes	Yes	Unk
19	CAR/DS Authoring Tool	CAR/DS	Yes	Radiology	Yes	Lung-RADS, BI-RADS, LI-RADS, Incidental Findings	Yes	No	Yes	No	Yes	No
20	SmartCare	OWL	Yes	Anesthesia	Yes	Physiological monitoring	Yes	No	Yes	Yes	Yes	Unk
21	PROforma Authoring Tool	PROforma, GLIF	Unk	Domain Indep	Yes	Routine prescribing system, pain control system, a system for advising medication for patients	Yes	No	Yes	Unk	Yes	Unk
22	OpenCDS	Arden Syntax	No	Domain Indep	Yes	Some use cases include pregnancy test and HIV test	Yes	Unk	Yes	Yes	Yes	Unk
23	OpenClinical	PROforma	Unk	Domain Indep	Yes	Head injury, COVID 19, Stroke	Yes	No	Yes	Yes	Yes	Yes
24	Rule Editor	Unk	Unk	Domain Indep	Yes	Medication reminder, health maintenance	Yes	No	Yes	Unk	Yes	Unk
25	SAGE Authoring Environment	SAGE	Unk	Domain Indep	Yes	Hypertension guideline	Yes	Unk	Yes	Yes	Yes	Yes
26	SAGEDesktop	SAGE	Yes	Domain Indep	Yes	Immunizations, diabetes	Yes	No	Yes	Yes	Yes	Yes
27	SAGE	OWL	Unk	Domain Indep	Unk	Computerized physician order entry (CPOE)	Yes	Unk	Yes	Yes	Yes	Unk
28	Smart CDSS Authoring tool	Arden Syntax	Yes	Domain Indep	Yes	Head and Neck Cancer diagnosis and treatment recommendations.	Yes	No	Yes	Yes	Yes	Yes
29	Protégé	OWL	Yes	Domain Indep	Yes	Unk	Yes	No	Yes	Yes	Yes	Yes
30	Internet Portal	Unk	Unk	Domain Indep	Yes	Unk	No	Unk	Yes	Unk	Yes	Unk
31	GEM Cutter	OWL, GLIF	Yes	Domain Indep	Yes	CPOE, asthma, obesity	Yes	No	Yes	Unk	Yes	Yes
32	GLARE	OWL	Yes	Domain Indep	Yes	Different phenomena, including bladder cancer, reflux esophagitis, heart failure, and ischemic stroke.	Yes	No	Yes	Yes	Yes	Yes
33	SmartCare	OWL	Yes	Anesthesia	Yes	Physiological monitoring	Yes	No	Yes	Yes	Yes	Unk

Abbreviations: ACE: Attempto Controlled English; CAR/DS: Computer-Assisted Reporting and Decision Support; CKAT: Clinical Knowledge Authoring Tool; CQL: Clinical Quality Language; GEM: Guideline Elements Model; GLIF: Guideline Interchange Format; HeD: Health eDecisions; OWL: Web Ontology Language; PROforma: Proformalisation (of medical knowledge); RDF: Resource Description Framework; SAGE: Standards-based Sharable Active Guideline Environment; SPARQL: SPARQL (Simple) Protocol and RDF Query Language; SWRL: Semantic Web Rule Language; Unk: Unknown.

Figure 3.

Summary of CKAT characteristics. CKAT: Clinical Knowledge Authoring Tool.

The reviewed articles included different user types of CKATs. After reviewing all CKATs, the following types of users were identified as potential CKAT users: (1) Subject Matter Experts: SMEs are CDS experts who know the best practices and the clinical setting. SMEs are typically a qualified healthcare informatics person specializes in CDS. (2) Clinical Experts: CEs have in-depth clinical knowledge about the subject on which CDS is authoring. CEs are typically the CPG authors. (3) Developers/Data Scientists: Technical experts who know how to encode the knowledge into a machine-readable format with assistance from SMEs and CEs. (4) Clinical Champion: CCs are the lead clinical experts in charge of the CBK model and CDS governance. (5) Guideline Developer: Technical developers converting CBK into knowledge base artifacts. (6) Administrators: Persons responsible for publishing and validating the CBK model in the clinical setting. The administrator is also in charge of data capture to assess the impact and performance of the CDS system.

The reviewed studies included different approaches to integrate CKATs in the CDS development workflow. After merging workflows of CKATs described in different articles, we identified the following shared components of knowledge management across CKATs: assembling, authoring, reviewing, testing, publishing, validating, and assessing the knowledge (Figure  4).

Figure 4.

Development life cycle of a computable knowledge model within CKATs. CKAT: Clinical Knowledge Authoring Tool.

DISCUSSION

CKATs are integral to the development and maintenance of CDS systems. CKATs enable authors to generate accurate, complete, and reliable digital knowledge artifacts in a relatively efficient and affordable manner. Although extensive studies have reviewed the effectiveness of CDS systems, research is lacking on the types and specifications of CKATs. To address the need for a list of CKAT features, this study aimed to review and compare knowledge authoring tools and derive the common features of CKATs that are published in peer-reviewed publications.

We identified 33 unique CKATs across 47 publications. More than half of published CKATs use open source software and close to 70% use a standardized API, hence providing an opportunity to integrate CKATs in various CDS systems. Most CKAT developers have attempted to increase the usability of their applications, with 94% describing the information flow, 97% providing a graphical user interface, and 97% offering intellisense features for coding knowledge models.

CKATs assessed in peer-reviewed publications are still immature in supporting enterprise level features that are needed for healthcare settings to develop, maintain, and deploy knowledge models over an extended period. For example, only 48% of the CKATs have been continuously deployed and assessed in clinical settings. Furthermore, team-based knowledge management, key for deployment in healthcare settings, is still lacking among published CKATs with only 36% of them offering a knowledge version control and 18% providing an approach to grade the knowledge, despite the fact that 70% of them are providing collaborative tools for knowledge authoring. These challenges have led most peer-reviewed CKATs to remain in limited use within academic settings. Moreover, additional work is needed to develop CKATs that can be seamlessly integrated with rapidly evolving health IT platforms such as EHRs.

Given the frequent changes of clinical practice guidelines, especially during public health emergencies such as the COVID-19 pandemic, CKATs should also offer more automated features to incorporate up-to-date knowledge from both clinical and public health sources. Although public health decision support systems are differentiated from CDS systems, CKATs are needed to author, revise, maintain, and update population-level knowledge models. Additionally, primary care settings, which often use preventative CDS systems, will benefit from merging existing public health guidelines into local CDS systems, especially when dealing with public health emergencies., Consequently, CKATs should support not only the curation and maintenance of clinical knowledge but also the creation and management of population and public health knowledge models.

Several ongoing and significant health informatics challenges were not addressed in the reviewed CKAT publications. None of these publications explained how CKATs, and their knowledge models, handle the data quality issues with EHR data., Using alternate or additional clinical data sources such as insurance claims, and how such data sources may affect the knowledge models, was also absent in the CKAT publications. For example, medication records in EHRs are prescriptions while insurance claims include medication re/fills thus conveying different meanings for knowledge models using such information. Another important issue not mentioned in the CKAT publications was the incorporation of nonclinical data sources such as social determinants of health (SDOH) in knowledge models. Individual and neighborhood level SDOH data are increasingly used in the clinical decision-making process to improve outcomes and reduce utilization., However, none of the reviewed CKATs mentioned how such unstandardized information would be encoded and integrated into the knowledge creation process.

Future research and development in CKATs should address multiple dimensions of the knowledge authoring process. CKATs should ease the authoring and reviewing of the knowledge rules. CKATs should further facilitate multiuser collaboration for knowledge development. Automating the CDS testing process and supporting standardized terminology systems are also essential for future CKAT development. CKATs should continue offering intellisense features for knowledge coding and providing a clinical simulation environment to increase the usability of such tools. CKATs should also offer continuous deployment and publishing capabilities while increasing/improving knowledge management features. Finally, CKATs should be assessed and validated along with CDS systems so that their effectiveness can be measured in the larger context of decision support. See the Supplementary Appendix for additional recommendations generated based on our review to enhance future research and development in CKATs.

Despite our valuable findings, this review has several limitations. First, we conducted a scoping review of literature, and not a systematic review, hence some CKATs may have been missed. Second, the review only included published peer-review publications. Therefore, CKATs lacking such publications (eg, commercial CKATs) are not presented in this review. Third, all CKAT features extracted and presented in this review are limited to information included in the peer-reviewed publications. The actual CKATs were not downloaded and assessed separately. Accordingly, features that may exist in a CKAT, but not reported in the publications, are not listed in this review. Finally, this review was limited to systems primarily designed as CKATs; and excluded tools that are primarily designed for analytical or machine learning purposes. As the gap between knowledge generation and knowledge authoring is closing by such tools, additional reviews are needed to assess the role of analytical tools as CKATs.

CONCLUSION

CKATs play an integral role in improving CDS systems. Our scoping review highlights the key specifications for a CKAT. The CKAT specification proposed in this review can guide CDS authors in developing more targeted CKATs.

AUTHOR CONTRIBUTIONS

SSN, PN, HL and HK designed the study. SSN, CL, and RD conducted the study including data collection and data analysis. HK led the review strategy, and guided the study team during the preparation of this manuscript. SSN prepared the manuscript draft which got reviewed and revisted with important intellectual input from PN, HL, and HK. All authors have complete access to the study data.

SUPPLEMENTARY MATERIAL

Supplementary material is available at JAMIA Open online.

Click here for additional data file.