Design, effectiveness, and economic outcomes of contemporary chronic disease clinical decision support systems: a systematic review and meta-analysis.

OBJECTIVES: Electronic health record-based clinical decision support (CDS) has the potential to improve health outcomes. This systematic review investigates the design, effectiveness, and economic outcomes of CDS targeting several common chronic diseases.
MATERIAL AND METHODS: We conducted a search in PubMed (Medline), EBSCOHOST (CINAHL, APA PsychInfo, EconLit), and Web of Science. We limited the search to studies from 2011 to 2021. Studies were included if the CDS was electronic health record-based and targeted one or more of the following chronic diseases: cardiovascular disease, diabetes, chronic kidney disease, hypertension, and hypercholesterolemia. Studies with effectiveness or economic outcomes were considered for inclusion, and a meta-analysis was conducted.
RESULTS: The review included 76 studies with effectiveness outcomes and 9 with economic outcomes. Of the effectiveness studies, 63% described a positive outcome that favored the CDS intervention group. However, meta-analysis demonstrated that effect sizes were heterogenous and small, with limited clinical and statistical significance. Of the economic studies, most full economic evaluations (n = 5) used a modeled analysis approach. Cost-effectiveness of CDS varied widely between studies, with an estimated incremental cost-effectiveness ratio ranging between USD$2192 to USD$151 955 per QALY.
CONCLUSION: We summarize contemporary chronic disease CDS designs and evaluation results. The effectiveness and cost-effectiveness results for CDS interventions are highly heterogeneous, likely due to differences in implementation context and evaluation methodology. Improved quality of reporting, particularly from modeled economic evaluations, would assist decision makers to better interpret and utilize results from these primary research studies. REGISTRATION: PROSPERO (CRD42020203716).

INTRODUCTION

Background and significance

Clinical decision support (CDS) is a process of using relevant clinical knowledge, or intelligently filtered patient information, to enhance health-related decisions. A broad definition of CDS includes both manual and computerized interventions. However, with the wide adoption of electronic health record (EHR) use in recent decades, contemporary CDS systems are increasingly computerized and data-driven. CDS builds upon existing EHR functionality, whereby algorithms are used to process EHR data, and individual-specific decision support is presented to clinician or patient end-users., CDS functions may be embedded into EHRs or exist as standalone applications.

Chronic diseases are the leading cause of death worldwide. Chronic disease CDS systems can target the whole continuum of clinical care from screening to diagnosis, to treatment, and follow-up. For example, CDS for diabetes can screen for the presence of disease based on laboratory results, improve diagnosis documentation, recommend appropriate medications or nonpharmacological management, and prompt adherence to guideline-based cycle of care. CDS holds great potential for improving chronic disease health care and outcomes, but what evidence exists to demonstrate CDS effectiveness?

The first systematic reviews of CDS interventions emerged in the late 1990, and a number of earlier reviews (including articles up to 2011) were conducted with a broad chronic disease or all-disease focus. Among these is a landmark systematic review by Bright et al, commissioned by the US Agency of Health Research and Quality; this review included 148 full texts of randomized controlled trials (RCTs), and examined the effectiveness of CDS systems across all medical fields, spanning primary research articles from 1976 to 2011. With the expansion of CDS literature over the past decade, recent CDS systematic reviews (including articles from 2011 onwards) have grown increasingly specialized. In particular, a large number of recent reviews have exclusively focused on medication-related CDS interventions. Other examples of recent CDS systematic reviews with a specific scope include: reviews that focus on a single disease (eg, diabetes); reviews of CDS with a focus on specific end-users (eg, nursing staff); and reviews that target a specific CDS type—such as laboratory test-related CDS,,, or diagnostic image-related CDS., In contrast to these systematic reviews, few recent reviews have had a broad chronic disease or all-disease focus—significantly, of these recent reviews, only 2 have explicitly excluded outdated computerized CDS systems that are not EHR-integrated., Thus our review sought to investigate the effectiveness of contemporary, EHR-based CDS interventions, with a broad chronic disease focus. Our review scope is most similar to El Asmar et al’s 2021 review with a focus on EHR-based CDS for chronic diseases—however, the authors of this review had a limited number of outcomes of interest, and only included a small number of studies (n = 8).,

Even where CDS interventions are effective, investment into such interventions is likely to depend on economic considerations. The cost of new health technologies and unsustainable rise in healthcare expenditure is a concern for economies worldwide; additionally, since the 2020 COVID-19 pandemic, OECD countries have seen a sharp rise in the ratio of healthcare expenditure to gross domestic product. CDS interventions require substantial upfront investment and in the face of competing healthcare priorities, weak financial business cases have been highlighted as a major macrolevel barrier for widespread CDS adoption. On the other hand, CDS interventions can also result in cost savings—for example, from averting adverse events, reducing unnecessary treatment, or reducing cost of downstream disease complications.,, While several systematic reviews of CDS effectiveness have reported costs alongside other health outcomes,,,,, there are few examples of comprehensive economic systematic reviews.,,, Given the high importance of financial considerations to policy makers, we included a detailed economic review alongside the effectiveness review.

Study objectives

CDS technology is continuously evolving and there remains a need to summarize up-to-date evidence of clinical and economic outcomes. To address this need, we conducted a systematic review to summarize the effectiveness and economic outcomes of contemporary EHR-based CDS interventions in cardiovascular disease, diabetes, chronic kidney disease, hypertension, and hypercholesterolemia. We selected these chronic diseases as they are common, share modifiable risk factors, and often exist as comorbid conditions., Our specific research questions are outlined below:

CDS design characteristics:

What are the design characteristics of contemporary EHR-based chronic disease CDS systems?

What clinical task do the CDS systems address, what EHR data types are used, and how does the CDS interact with users?

Effectiveness:

Are CDS interventions effective compared to control groups in improving chronic disease care?

Is there evidence for longer-term sustained effectiveness (beyond 12 months)?

Are certain CDS features more likely to result in positive effectiveness outcomes?

Economic:

What are the costs and cost-effectiveness of CDS systems compared to usual care?

What economic evaluation methods are used in the analyses?

MATERIALS AND METHODS

This systematic review was registered on PROSPERO (CRD42020203716). The overall approach to the review was a mixed methods study summarizing the effectiveness, economic, and qualitative outcomes of CDS for use in chronic disease management. Effectiveness and economic reviews are included in this paper and the qualitative component will be reported separately. JBI methods for review of effectiveness and economic evidence informed our methods. The review adhered to Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines.

Inclusions and exclusions

The population of interest was patients with one or more of the following chronic diseases: cardiovascular disease, diabetes, chronic kidney disease, hypertension, and hypercholesterolemia. Our intervention of interest was EHR-based CDS used in nonacute settings, including general practice and specialist outpatient settings. Comparison groups included a control group, historic controls, or usual care. The outcome of interest included measures of clinical effectiveness (eg. morbidity, mortality) or health process outcomes (eg. guideline adherence). The outcomes of interest for the economic component of the study were costs and cost-effectiveness. Relevant study designs included RCTs, quasi-experimental studies, partial economic evaluations (cost analysis), and full economic evaluations. The landmark systematic review by Bright et al and at least a dozen other broad systematic reviews adequately covered the period up to January 2011;, thus, we restricted our search to a 10-year window between January 2011 and January 2021.

Exclusions consisted of CDS targeting populations with acute conditions (eg, acute heart failure) or targeting chronic disease management in acute settings (eg, perioperative diabetes blood sugar management). Computerized CDS interventions not linked to an individual’s EHR (eg, web-based cardiovascular risk calculator), or those that performed basic EHR functions (eg, simple recall functions) were excluded. In terms of article type, we excluded EHR-based CDS prototypes if they were not implemented in a clinical setting. Nonprimary research studies and non-English language studies were also excluded. For our full inclusion and exclusion criteria, see Supplementary Appendix SI.

Search strategy

We searched PubMed (Medline), EBSCOHOST (CINAHL, APA PsychInfo, EconLit), and Web of Science databases. PubMed MeSH terms and title/abstract search terms included concepts of “clinical decision support systems,” “cardiovascular disease,” “diabetes,” “chronic kidney disease,” “hypertension,” “hypercholesterolemia,” and were translated into the other included databases. A research librarian provided input into the search strategy. See Supplementary Appendix SII for the full PubMed search strategy.

Study selection and data extraction

Abstracts and titles were screened in duplicate by 2 independent reviewers (WC, and BB or PC) using the Covidence software (Veritas Health Innovation, Melbourne, Victoria, Australia). Full texts were verified against the inclusion and exclusion criteria, and classified into effectiveness, economic, or qualitative outcome categories by a single reviewer (WC). Reasons for exclusion at full text stage were recorded. Citations were then imported into JBI System for the Unified Management, Assessment and Review of Information (JBI SUMARI) (JBI, Adelaide, Australia) for critical appraisal, data extraction, and meta-analysis.

Data extraction was conducted in duplicate by 2 reviewers (WC, and CO or PC). Study characteristics were extracted using the JBI SUMARI forms for effectiveness and economic studies. Additional standardized data forms were created to extract details of CDS characteristics and economic evaluation methods. Several dimensions of CDS design features were recorded: (1) clinical task addressed by the CDS (eg, CDS for screening); (2) EHR data types used in generating the decision support; and (3) CDS user interface features using a modified classification from Osheroff et al. We classified CDS types in effectiveness studies only—this is because CDS descriptions in economic studies are limited and typically reference a previously published effectiveness study. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Checklist informed our economic evaluation data extraction form. Details extracted included vehicle of analysis (trial or modeled), and methods for estimating costs and outcomes. Authors were contacted for further information where economic evaluation methods were limited in the primary research article.

Methodological quality assessment

Quality assessment was conducted by 2 reviewers (WC, and CO, KH, or GG) using the JBI critical appraisal tool for randomized control studies, quasi-experimental studies, and economic evaluations. To ensure consistency, a random sample of 25% of studies was independently assessed in duplicate by 2 reviewers. Discrepancies at all stages of the review (screening, data extraction, and methodological quality assessment) were resolved by consensus and where necessary, by a third team member. Our review aimed to summarize all relevant CDS studies in the field; therefore, studies were not excluded from data extraction or synthesis based on quality assessment scores.

Analysis

Descriptive statistics were reported for study characteristics and CDS design characteristics. A meta-analysis was conducted to summarize similar effectiveness outcomes. The meta-analysis included results from RCTs only due to inconsistent study design, lack of similar effectiveness outcomes, and variable reporting methods in quasi-experimental studies. The meta-analysis was conducted with the primary objective of providing a visual (forest plot) and quantitative summary of existing literature, rather than providing an estimate of effect size for CDS interventions. We used a random effects model; examined effect sizes in terms of relative risks (RR) for categorical variables and mean difference for continuous variables; and reported the I2 statistic to describe study heterogeneity. Heterogeneity of effectiveness results was investigated with an exploratory meta-regression analysis using R (R Core Team 2021)—univariate and multivariate logistic regression (using a full-model approach) were conducted to examine the presence of associations between CDS features and a primary positive outcome.

A narrative review was conducted for economic studies. Key economic evaluation methods and results are outlined in a summary table and costs are converted to USD 2021 prices to enable comparison. Costs in the original currency were inflated to 2021 values, then converted to USD using 2021 purchasing power parity conversion factors.

RESULTS

Study selection

The search of databases yielded 14 422 citations and 6423 duplicate records were removed (Figure 1). Of the 7999 articles undergoing abstract and title screening, 625 were eligible for full text screen. Articles were excluded from full text screening primarily due to absence of a relevant chronic disease CDS systems mentioned in the title or abstract. Of the full text articles screened, 377 were excluded based on article type (eg, protocol or other nonprimary research articles) and 148 were excluded based on article content. Common reasons for exclusion based on article content included CDS intervention lacking an EHR component, or EHR interventions lacking a CDS component. A total of 76 studies met the criteria for inclusion in the effectiveness review, and 9 studies met the criteria for inclusion in the economic review.,,,,, Five studies had both effectiveness and economic outcomes, and were therefore included in the article count for both reviews. Qualitative outcomes studies (n = 33) of the systematic review are reported separately. Of the effectiveness studies, 32 were RCTs and 44 were quasi-experimental studies. Thirty-one of the 32 RCTs had a suitable outcome for inclusion in the meta-analysis of effectiveness outcomes.

Figure 1.

PRISMA flow diagram.

The number of articles in the field has increased substantially since 2011. Figure 2 reveals a doubling of articles identified using our PubMed search strategy from the past decade (January 2011 to January 2021), compared to the number of articles present prior to 2011.

Figure 2.

Count of PubMed search results (January 1972 to January 2021).

Methodological quality

Methodological quality was moderate for RCT and quasi-experimental studies. The main source of bias in RCTs arose from the inability to blind healthcare providers or assessors (JBI checklist for RCTs Q5 and Q6). For quasi-experimental studies, the main methodology quality issues were that less than half of all studies demonstrated a similar comparison group (JBI checklist for quasi-experimental studies Q2), and that few studies had multiple measurements of outcomes pre- and postintervention (Q5). Economic evaluation studies were of limited methodological quality. Most studies had unclear descriptions of the “usual care” comparison group (JBI checklist for economic evaluations Q2), and limited descriptions of how identification and valuation of costs and outcomes were conducted (Q3, Q5, Q6). See Supplementary Appendix SIII for the methodological quality of included studies.

Characteristics of included studies

Study design of effectiveness studies included 32 RCTs and 44 quasi-experimental studies. RCTs were predominantly clustered RCTs randomized at a practice or practitioner level (n = 26), with few RCTs being randomized at an individual level. For quasi-experimental studies 32 were before-after studies and 12 included a nonrandomized control group. Nonrandomized control groups include self-selection into a control group (natural experiment), or allocation without randomization into a control group. See Supplementary Appendix SIV for characteristics of included studies and full reference list of included studies.

Table 1 presents an overview of study characteristics and settings. Most studies (88%) were conducted in high income countries, with half of all studies (50%) conducted in the United States. Several studies from low- and middle-income countries were also included (12%). The top 3 diseases addressed by CDS interventions were multiple cardiovascular risk factors (25%), diabetes (18%), and chronic kidney disease (13%). Included studies were predominantly conducted in primary care settings (70%). In the United States, “primary care” referred to primary care practices staffed by family physicians, internal medicine physicians, and supporting clinicians; in other countries, primary care referred to general practice or community health clinics. Forty-one percent of studies had additional interventions bundled alongside the CDS intervention—these interventions included additional education, audit, or other quality-improvement initiatives, individualized feedback, and financial incentives. Start dates for CDS studies included in this analysis ranged from 2001 to 2017 with a median study duration of 12 months. There was a median lag-time of 4 years between study commencement and year of publication (Supplementary Appendix SVI and Figure S2).

Table 1.

Characteristics and settings of included studies

Study characteristic	Effectiveness articles (n)	% (total n = 76)	Economic articlesa (n)	% (total n = 9)
Country
USA	38	50	2	22
Australia	6	8	2	22
India	5	7	1	11
UK	5	7	1	11
Canada	3	4	2	22
South Korea	2	3	0	0
Sweden	2	3	0	0
Italy	2	3	0	0
Belgium	2	3	0	0
Netherlands	2	3	0	0
Multiple countries	2	3	0	0
Other	5	7	1	11
CDS disease focus
Cardiovascular risk factors	19	25	3	33
Diabetes	14	18	2	22
Chronic kidney disease	10	13	1	11
Multiple other	8	11	0	0
Atrial fibrillation	8	11	1	11
Hypertension	7	9	0	0
Vascular conditions	4	5	0	0
Other	6	8	2	22
Study duration
≤6 months	17	22	N/Ab	N/A
6–12 months	33	43	N/A	N/A
>12 months	24	32	N/A	N/A
Study setting
Primary care—other	31	41	3	33
Primary care—general practice	22	29	4	44
Multiple settings	13	17	1	11
Specialist outpatients	8	11	0	0
Other	2	3	1	11
Clinics/practices
≤25	47	62	5	56
25–50	6	8	1	11
50–100	10	13	2	22
>100	5	7	1	11

Note: Studies with missing data (eg, no study duration reported) are not included in this table.

Five of the 9 economic studies had both effectiveness and economic outcomes and were included in both reviews.

See Table 3 for time horizon of economic studies.

Table 3.

Economic evaluation methods and main results

	Study type	Study vehicle	Model type	Perspective	Time horizon (years)	Discount rate (%)	Cost—CDS intervention			Cost—healthcare costs			Sensitivity analysis		ICER CDS vs control in original currency	ICER CDS vs control in USD (2021 prices)
							Development	Implementation	Maintenance	Outpatient	Inpatient	Medication	One-way	PSA
Anchala 2015	CEA	Trial-based	N/A	Healthcare system	1	3	+	+	Unclear	+	−	+	+	N/A	Not reported
Gilmer 2012	CUA	Modeled	Markov microsimulation	Healthcare system	40	3	−	+	+	+	+	+	+	+	USD$3017 per QALY	USD$3811 per QALY
Orchard 2020	CUA	Modeled	Not specified	Healthcare funder	10	5	−	+	−	Unclear	+	+	+	−	AUD$16 578 per QALY, AUD$84 383 per stroke	USD$11 747 per QALY, USD$59 792 per stroke
O’Reilly 2012	CUA	Modeled	Markov microsimulation	Healthcare funder	40	5	+	+	−	+	+	+	+	−	CAD$160 845 per QALY	USD$151 955 per QALY
Oxendine 2014	Cost analysis	N/A	N/A	Local facility	N/A	N/A	−	−	−	−	+	−	N/A	N/A	N/A
Patel 2020	CEA	Modeled	Not specified	Healthcare system	5	3	−	+	+	−	+	+	+	−	AUD$7406 per primary CVD event prevented, AUD$17 988 per secondary CVD event prevented	USD$5248 per primary event, USD$12 746 per secondary event prevented
Ranta 2015	Cost analysis	N/A	N/A	Healthcare system	N/A	N/A	−	−	+	+	+	+	N/A	N/A	N/A
Subramanian 2012	Cost analysis	N/A	N/A	Local facility	N/A	N/A	−	−	−	−	−	+	N/A	N/A	N/A
Willis 2020	CUA	Modeled	Markov cohort	Healthcare funder	Lifetime horizon	3.5	+	+	−	+	+	+	+	+	GBP£1359 per QALY for risky prescribing module	USD$2192 per QALY
Total (Yes—n)							3	6	3	5	7	8	6	2
Total (Yes—%)							33	67	33	56	78	89	67	22

CEA: Cost Effectiveness Analysis; CUA: Cost Utility Analysis; CVD: Cardiovascular Disease; QALY: Quality-Adjusted Life Years; PSA: Probabilistic Sensitivity Analysis.

Characteristics of CDS systems

A summary of CDS design characteristics is displayed in Figure 3. Definitions and examples of our CDS classification are outlined in Table 2. The majority of CDS systems were computer-based, with several studies reporting mobile- or tablet-based systems (n = 6). The most common clinical task addressed was pharmacological management (71%). Few individual CDS system (9%) addressed all aspects of chronic disease care (screening/diagnosis, pathway, pharmacological, and nonpharmacological management). A variety of EHR data types were used to generate decision support recommendations, with a median of 4 EHR data types used per individual CDS system. Alerts and pop-ups were the most common CDS user interface feature (89%), but most CDS interventions used 2 or more different user interface features (83%). See Supplementary Appendix SV for CDS design characteristics classification. See also Supplementary Appendix SVI for other additional figures and tables.

Figure 3.

Characteristics of clinical decision support systems.

Table 2.

Definitions and examples of clinical decision support classifications

CDS clinical task addressed	Definition and/or examples
Prevention/Diagnosis	Screening for disease or likelihood of disease (eg, cardiovascular risk screening), and diagnosis (eg, documentation of hypertension diagnosis)
Pathway	Referral for investigations (eg, bloods), referral to specialist, and other care pathway tasks
Pharmacological	Prescribing, dosing, and other medication changes
Nonpharmacological management	Patient education (eg, via patient dashboard), diet and exercise recommendations, and other nonpharmacological management (eg, smoking cessation)
EHR data types used	Definition and/or examples
Demographics	Age, sex, and other demographic data
Diagnosis	Checks for existing coded diagnosis within EHR (eg, displays alert based on existing diagnosis of atrial fibrillation)
Medication	Medications within EHR (eg, uses presence or absence of ACE-inhibitors to generate a decision support in chronic kidney disease prescribing)
Observation	Structured EHR data for observations (eg, systolic blood pressure readings, body mass index)
Laboratory	Structured EHR data for laboratory results (eg, HbA1c, urine albumin–creatinine ratio)
Manual entry	Use of additional manual data entry for CDS to generate decision (eg, family history, depression scale)
CDS user interface features	Definition and/or examples
Form/template	Provides auto-fill pathology (order set), imaging templates, automated specialist referrals
Data presentation—written summary	Displays written patient summaries, primarily text-based (eg, one-page patient summary with recent results)
Data presentation—visual summary	Displays visual patient summaries, primarily graphics-based (eg, dashboard with dial, traffic light systems)
Data presentation—risk scoring	Provides risk scores displayed in numerical, color, or other format (eg, risk scoring for atrial fibrillation to aid with prescribing decisions)
Prescribing/dosing	Provides recommendations or tools for prescribing, dosing, and other medication changes
Pathway support	Provides cycle of care pathways, checklists for periodic visits, and other follow-up support
Reference info	Provides general or patient-specific knowledge resources (eg, Info Buttons, links to relevant guidelines)
Alerts/reminder	Displays alerts and reminders (eg, pop-up to identify at risk patients, red alert for incorrect drug dosing)
Service/population-level summary	Provides overview of patients and assists with quality improvement at a service or population level (eg, disease registry, service-level tools)

ACE-inhibitor: Angiotensin Converting Enzyme Inhibitor; EHR: Electronic Health Record; HbA1c: Hemoglobin A1c.

Findings of effectiveness studies

Studies reported a mix of health process (78%), clinical (70%), and/or user outcomes (5%). Overall, 63% of studies described a positive primary trial outcome that favored the CDS group compared to controls; the remaining studies reported equivocal primary outcomes. Results from 31 RCTs were included in the meta-analysis. One RCT was excluded from the meta-analysis because it did not include a similar effectiveness outcome for synthesis.

Figure 4 is a forest plot showing relative risk (RR) of the proportion of patients meeting study-determined clinical targets for CDS versus control groups (HbA1c, SBP, LDLc). Figure 5 shows the absolute mean differences in clinical outcomes. Meta-analysis of other clinical and process outcomes (eg, proportion of patients meeting guideline adherence) is reported in Supplementary Appendix SVI. Results of our meta-analysis demonstrated small improvements in outcomes in CDS versus control, but these were generally of limited clinical and statistical significance. High levels of heterogeneity existed among the studies, particularly in relative risks (RR) of outcomes between the studies with I2 ranging between 78 and 85 (Figure 4). Effectiveness outcomes were commonly reported up to a 12-month timeframe (66%) and none of the RCTs were conducted beyond this 12-month timeframe. Thus, the sustained longer-term effects of chronic disease CDS interventions remain unclear. An exploratory meta-regression did not reveal CDS design features that were clear and statistically significant predictors of a positive primary outcome (Supplementary Appendix SVI and Figures S2 and S3).

Figure 4.

Meta-analysis of clinical outcomes (relative risks).

Figure 5.

Meta-analysis of clinical outcomes (mean differences).

Findings of economic evaluation studies

Study methodology

Table 3 provides an overview of economic evaluation methodology and findings. Detailed economic evaluation methods for each included study are outlined in Supplementary Appendix SVII. Full economic evaluation was conducted in 6 studies and cost analysis was conducted in 3 studies. Most studies used a healthcare system or healthcare funder perspective (n = 7). Five out of 6 economic evaluations used modeled approaches—Markov models were the most common approach and model type was unclear in the remaining 2 modeled studies. Time horizon ranged from 1 year in the trial-based study, up to 40 years in modeled studies. Discount rate per annum ranged from 3% to 5%.

Costs and outcomes

Resource utilization was measured with a combination of microcosting and aggregate costing approaches. Valuation of unit costs came from both trial-based data and published country-specific data. Implementation cost was the most common type of CDS intervention cost included (67%) and medication costs was the most common type of healthcare cost included (89%).

CDS intervention costs were the main driver of the overall incremental cost differences between CDS versus control groups. The highest CDS intervention cost was reported by Willis et al at USD$3934 per patient per year. In contrast, the relative contribution of healthcare costs to overall incremental cost differences was minor—healthcare costs in the CDS groups ranged from a maximum of an additional USD$11per patient per year, to cost savings of up to USD$25 per patient per year. Increased healthcare cost in the CDS group was primarily attributable to increased medication costs. Conversely, cost savings in the CDS group arose from decreased downstream disease management and hospitalization costs.

For modeled studies, short-term trial-based outcomes were extrapolated to long-term outcomes. Short-term outcomes in all studies consisted of clinical differences in SBP, HbA1c, or LDLc at 12 months. The effect sizes of these short-term outcomes were small—of note, short-term changes were included in modeled analyses, regardless of whether clinical improvements were statistically significant. Modeled long-term outcomes included morbidity and mortality outcomes (eg, cardiovascular event and cardiovascular death). Method for valuation of health states, that is, sources of utility weights used to calculate QALYs, was not described in any of the included studies.

Cost-effectiveness and sensitivity analysis

The reported incremental cost-effectiveness ratios (ICERs) ranged widely from a minimum of USD$2192 per QALY, to $USD151 955 per QALY. One-way sensitivity analysis was conducted in all 6 full economic evaluations but probabilistic sensitivity analysis was conducted in only 2 out of the 5 modeled studies.

DISCUSSION

Main findings

The volume of literature on CDS interventions for chronic diseases has risen substantially over the past decade. This review contributes to our understanding of CDS systems in several ways. Firstly, we provide an up-to-date summary of CDS effectiveness, across a broad range of chronic disease CDS interventions. Secondly, we provide insight into the design characteristics of contemporary, EHR-based chronic disease CDS systems. Many authors have attributed heterogeneity in meta-analysis results to differences within individual CDS systems—yet, surprisingly little has been published alongside reviews to systematically categorize CDS design features. For example, Kwan et al recently conducted a recent meta-analysis of CDS effectiveness studies; while comprehensive in terms of EHR-based CDS types included, the authors’ description of individual CDS type was mostly limited to user interface features (eg, interruptive vs noninterruptive). Taxonomies of CDS systems have been published but are rarely used in full. We simplified a previous review’s approach to extracting CDS design characteristics and provide a clinically relevant insight into what each CDS does, how they work (EHR data types used in CDS algorithms), and how decisions are communicated to users. Thirdly, our economic review not only summarizes economic outcomes, but also critiques evaluation methods and economic models used—this body of work is increasingly relevant as the number of modeled studies in the field exceeds that of trial-based studies.

It is encouraging that many CDS interventions sought to tackle several related cardiometabolic comorbid conditions. However, as with a previous systematic review, we found that CDS systems mainly addressed an index condition of concern and poorly accounted for multimorbidity. Our analysis of CDS design features found that a narrow scope of EHR data types contributes to a lack of wide clinical applicability. For example, a CDS targeting abdominal aortic aneurysm screening may only use coded diagnosis for aortic aneurysm but requires additional manual data entry for existing comorbidity (eg, hypertension); or, a CDS that targets hypertension management may only extract EHR medication data for first line medication classes but not account for presence of related cardiac medications. Kawamoto and McDonald have called out this “mismatch” between information needed for decision rules and the lack of EHR data availability as a critical reason for CDS failure. An assortment of user interface features was described, with most CDS systems utilizing more than one user interface feature. However, many contemporary CDS designs are still centered around alerts and reminders. Alerts and reminders have been the main CDS type described since the late 1990s, and progress in CDS user interface design is slow. This is problematic as alert fatigue is a well described issue,—furthermore, poor usability can be a barrier to CDS uptake, and contribute to clinician burnout.

Several reviews to date have shown that CDS systems have a modest effect on improving health processes or morbidity, but not mortality.,, Results for CDS specifically targeting primary care or chronic disease settings has been less clear., Our meta-analysis found that CDS generally resulted in a favorable effectiveness outcome, but with small effect sizes of limited clinical and statistical significance. In our review, heterogeneity likely occurred due to differences in study design and diverse real-world contexts for CDS implementation. Meta-analyses of complex interventions, such as CDS interventions, is notoriously difficult to interpret., In CDS studies, varying effect sizes can be due to inconsistent CDS uptake despite availability of the intervention.

Previous meta-regressions have identified automated provision of decision support,,, CDS systems requiring reasons for override,, and other CDS success factors. Our meta-regression did not highlight CDS features associated with a primary positive outcome. The difference is unsurprising, as meta-regression results are highly dependent on study selection, as well as reviewer-defined abstraction of CDS factors. It is also possible that factors not considered in our meta-regression (eg, user factors, clinical champions) were more influential in determining CDS success than the CDS features extracted. Qualitative evaluation have an important role in explaining quantitative results—as such, a qualitative review has been conducted alongside this systematic review to further explore human and technological factors to CDS success.

Full economic evaluations are notoriously scarce in the CDS literature.,, Our economic review results are consistent with previous publications that describe inconclusive evidence for cost-effectiveness of CDS interventions.,,,, In a 2015 review, Jacob et al described ICERs of cardiovascular CDS interventions ranging widely from USD$16 500 to USD$162 000 per QALY. We found a similarly large variation in estimated cost-effectiveness, which was largely dependent on parameters selected for modeled analyses. As with previous reviews, we found that reporting quality was poor and hindered interpretation of economic results., For example, some authors reported an aggregate figure for incremental cost without specifying whether these costs were CDS or healthcare-related. Financial considerations are important in shaping resource allocation and reimbursement policies, but more work needs to be done to articulate the value proposition of CDS technologies.,,

Limitations

There are several limitations to our review. We considered common chronic diseases but did not include noncardiometabolic chronic diseases such as chronic obstructive pulmonary disease. Two reviewers conducted title and abstract screening, but full text screening was performed by a single reviewer. We did not exclude effectiveness and economic studies of lower methodological quality because our review aimed for breadth and generalizability. We used a comprehensive search query across several large databases, but did not include non-English articles and gray literature. Due to limited CDS descriptions within the text, data extraction for some CDS features were based on implicit information rather than explicit information found in the primary research articles. Over a third of CDS interventions were bundled together as part of an overall quality improvement initiative; furthermore, chronic disease interventions are implemented in the context of multimorbidity and health systems changes. Thus, effectiveness outcomes could not be attributed to the EHR-based CDS alone. Similarly, cost-effectiveness of the CDS was often unable to be separated from the cost-effectiveness of the overall intervention within the primary research (eg, overall screening initiative). The meta-analysis results need to be interpreted with caution given varied methodological quality in primary research studies and heterogeneity in the CDS interventions. Finally, we focused on contemporary EHR-based CDS studies published over the past decade but recognize that limiting our search from 2011 onwards does not guarantee that the most up-to-date CDS technology are included. We found that some of the included interventions (eg, alert-based CDS) are similar in nature to computerized CDS described in previous reviews of studies prior to 2011. Furthermore, a considerable lag-time exists between year of study commencement and year of publication, which limits our ability to capture the most up-to-date tools in our review.

Future directions

Learning from the collective experience of CDS design, development, implementation, and evaluation is key to achieving the full benefits of CDS technology. We provide 2 key recommendations for future CDS design and research. Firstly, despite decades of CDS research, many recent CDS interventions remained limited in scope—for example, a CDS may concentrate on hypertension screening without relevant tools to aid hypertension prescribing. Instead of building multiple CDS systems of narrow clinical scope, an ideal CDS systems would target screening, diagnosis, and management of multiple chronic diseases. A generalized chronic disease CDS would also require innovative user-interfaces that evolve beyond simple alerts to communicate decision support to users. Several authors have proposed problem orientated summaries—displaying relevant EHR information and decision support for each disease—as a promising solution for reducing cognitive load, and improving CDS usability in chronic disease settings., Nevertheless, such approaches depend on integration of a much wider variety of EHR data types than what is currently used; requires ongoing work toward technological standards, and needs greater subject matter expertise collaboration across clinical specialties., Secondly, we echo previous authors in recommending that CDS evaluations include sufficient details of the CDS technology itself., “Clinical decision support” is a broad term. During data extraction, we found that descriptions of CDS systems were limited. Authors should clearly describe CDS features—ideally through graphical representations (eg, screenshots of the live CDS interface), and make full use of Supplementary Materials to describe what their CDS systems entailed, and the degree to which it was EHR-enabled.

Our findings also have implications for CDS evaluation research. Chronic diseases impact individuals and health systems over time but evidence for CDS effectiveness beyond 12 months is unclear. Funding for CDS interventions tends to be small in comparison to drug and device trials and there remains a need to fund high quality, longer-term studies with meaningful evaluation measures., Modeled studies are increasingly used to project longer-term CDS costs and outcomes. While useful in enabling complexity in economic evaluations, to be credible and replicable, future economic modeling needs to move beyond a “black box” approach—that is, there needs to be greater transparency in the reporting of model types, key parameter differences between intervention and control groups, and description of parameter sources (eg, trial-based, expert opinion). Finally, both effectiveness and economic studies should better account for “e-iatrogenesis” in terms of unintended consequences associated with CDS use.,, In our review, the main adverse health outcome considered was bleeding events secondary to CDS interventions for anticoagulation. The clinical and economic impact of e-iatrogenesis, for example, from false positive CDS screening results and increased healthcare utilization, needs to be further considered in future evaluations.,

CONCLUSION

Our systematic review summarizes contemporary CDS systems used in chronic diseases and provides recommendations for future CDS designs. CDS interventions demonstrated small improvements to health or process outcomes, but the effect sizes tended to be of limited clinical and statistical significance. Evidence for cost-effectiveness was inconclusive with large variations in ICERs between studies. The diverse nature of CDS technologies, real-world implementation contexts, and evaluation methodologies all contributed to highly heterogeneous results. Future CDS evaluations should aim for higher quality reporting of CDS design characteristics and evaluation methods. Improved reporting would ensure that results from CDS evaluations are interpretable and useful for decision makers.

FUNDING

This research was supported by the Royal Australian College of General Practitioners (RACGP), the Australian Government Research Training Program (RTP) Scholarship, and Menzies School of Health Research scholarship.

AUTHOR CONTRIBUTIONS

WC, KH, AC, GG, and AA contributed to the conception and design of the study. WC conducted the search strategy. WC, BB, and PC screened titles and abstracts for eligibility. WC screened full texts for inclusion. WC, CO, KH, and GG conducted critical appraisal. WC, CO, and PC conducted data extraction. WC and PC analyzed the data. WC prepared the original draft. All authors reviewed the final manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Click here for additional data file.