An evaluation of clinical decision support tools for Patient Health Questionnaire-9 administration.

INTRODUCTION: Many health care institutions are working to improve depression screening and management with the use of the Patient Health Questionnaire 9 (PHQ-9). Clinical decision support (CDS) within the EHR is one strategy, but little is known about effective approaches to design or implement such CDS. The purpose of this study is to compare implementation outcomes of two versions of a CDS tool to improve PHQ-9 administration for patients with depression.
METHODS: This was a retrospective, observational study comparing two versions of a CDS. Version 1 interrupted clinician workflow, and version 2 did not interrupt workflow. Outcomes of interest included reach, adoption, and effectiveness. PHQ-9 administration was determined by chart review. Chi-square tests were used to evaluate associations between PHQ-9 administration with versions 1 and 2.
RESULTS: Version 1 resulted in PHQ-9 administration 77 times (15.3% of 504 unique encounters) compared with 49 times (9.8% of 502 unique encounters) with version 2 (P = .011). DISCUSSION: An interruptive CDS tool may be more effective at increasing PHQ-9 administration, but a noninterruptive CDS tool may be preferred to minimize alert fatigue despite a decrease in effectiveness.

Introduction

Depression affects nearly 300 million people worldwide and can significantly interfere with quality of life.1 The World Health Organization1 reports that depression is a significant cause for disability leading to emotional and physical challenges. Unfortunately, depression remains underdiagnosed and undertreated.2,3

To assist in diagnosis and treatment, there are many screening tools available. The most widely used tool for clinical care is the validated Patient Health Questionnaire-9 (PHQ-9), which has become best practice for identifying depression.4 The PHQ-9 is used to help diagnose depression and monitor symptom severity.4 To facilitate integration into clinician workflow and clinical documentation, PHQ-9 scores are often documented in structured fields within EHRs. However, despite the widespread availability of the PHQ-9, fewer than 3% of US adults are screened for depression during office visits.5

The Centers for Medicare and Medicaid Services6 and the US Preventive Services Task Force7 both recommend routine depression screening in adults and encourage use of the PHQ-9. Furthermore, in value-based reimbursement models, Centers for Medicare and Medicaid Services financially incentivizes practices to use the PHQ-9. As such, health care institutions are implementing interventions to improve test administration and documentation of PHQ-9 scores. One potential intervention is to incorporate clinical decision support (CDS) tools within EHRs. Here, we describe the design and implementation outcomes of the addition of a CDS tool to improve PHQ-9 administration in a primary care setting.

Methods

This is a retrospective study comparing implementation outcomes of two versions of a clinician-facing CDS tool to improve PHQ-9 administration. Version 1 was implemented in May 2016 for approximately 18 months. After the trial period of version 1, version 2 was designed with input from clinician end users and implemented in November 2017. Outcomes of the alerts from version 1 during the month of February 2017 were compared with outcomes of version 2 in February 2018. This study compares reach (patients and clinicians exposed), adoption (clinician did not dismiss), and effectiveness (documentation of PHQ-9 scores) of the 2 versions of the CDS tool from the perspective of the clinician.

Description of the Health System and CDS Tools

This study took place across 13 primary care practices within a large health system. The health system has used 1 integrated EHR, Epic Systems, since 2011.

The goal of both versions of the CDS tool was to increase PHQ-9 administration and documentation of scores when the most recent PHQ-9 score was elevated and a repeat PHQ-9 was clinically indicated. Version 1 was designed with input from a multidisciplinary team representing primacy care clinicians, psychiatry, clinical informaticians, analysts, and population health experts. It was designed as an active CDS that would interrupt clinicians (advanced practice providers, residents, physicians) each time they opened a clinical encounter for a patient meeting the following criteria: most recent PHQ-9 was ≥9 and last alert was ≥3 months ago. The alert that appeared recommended administration of the PHQ-9 and provided 8 response options (Table 1), including the option to update the PHQ-9 score or to indicate the PHQ-9 was not appropriate. Once a clinician selected 1 of the 8 response options, the CDS would no longer be visible for that encounter; otherwise, it would continue to interrupt whenever a provider entered the patient encounter.

TABLE 1

Design features of both versions of the clinical decision support (CDS) toola

Design Feature	CDS Version 1	CDS Version 2
Alert type	Active (interruptive, “pop up”); displays when opening patient chart	Passive (noninterruptive); accessible by clicking icon in banner or in clinical documentation
Frequency of alert	More than once per clinician during an encounter	Once per clinician during an encounter
Clinician response options	Many: Go to update PHQ-9* Cultural/language barrier Patient declined Not appropriate today Treated by psychiatrist PHQ-9 today Update problem Elevated PHQ-9 without a list* diagnosis of depression	Few: Will give PHQ-9 Go to update PHQ-9* Update problem list*
End user	Primary care clinicians	Primary care clinicians
Patient eligibility	Adults with most recent PHQ-9 score ≥9; 3 mo since the last alert	Adults with most recent PHQ-9 score ≥9 within last 18 mo; 3 mo since the last alert

PHQ-9 = Patient Health Questionnaire-9.

If a clinician selected 1 of these response options denoted by an asterisk, the clinician could also select 1 of the other response options. However, a clinician was not able to select more than 1 of the response options without an asterisk.

Interim evaluation of version 1, including clinician feedback, revealed that revisions were needed to address concerns of excessively frequent and sometimes inappropriate alerts. For example, it was identified that version 1 would sometimes alert more than once for a given clinician-patient visit (if a response option was not selected). Clinicians also stated that version 1 would sometimes alert when the most recent PHQ-9 was several years old and, thus, no longer relevant. During the interim evaluation, the clinician dismissal rate for version 1 was estimated to be 94%.

The redesigned CDS tool (version 2) incorporated the following changes: it was passive, only considered PHQ-9 scores within the past 18 months, and had a simplified user interface with fewer response options. As a passive tool, version 2 did not interrupt workflow, but was available on demand for clinicians through 2 different pathways: 1) by clicking on an icon in the top banner/header of the EHR screen or 2) navigating to the dedicated CDS section available when documenting clinical encounters. As a passive CDS, it was available throughout the patient encounter and allowed access to the information at the clinician's discretion. Table 1 compares the design specifications of the 2 CDS tools.

Parallel to implementation of versions 1 and 2 of the clinician-facing CDS tool, 2 versions of a medical assistant–facing CDS tool were also implemented. The medical assistant–facing CDS tool was implemented to support clinical workflows that encouraged medical assistants to administer PHQ-9 surveys to patients during the rooming process. Similar to the clinician-facing CDS tools, version 1 of the medical assistant–facing CDS tool was active with multiple response options and an estimated 99% dismissal rate. Version 2 of the medical assistant–facing CDS was subsequently designed to be passive with few response options.

Data Collection

Instances of the CDS alerts, the identity of the patient, and clinician responses to the CDS were collected using an analytics reporting tool within the EHR. Patient and visit characteristics were collected via manual chart review. PHQ-9 completion was reviewed via manual chart review as responses to a CDS are not always reflective of actual action taken. Completion of the PHQ-9 was defined as documentation within structured fields of the EHR. A 20% sample of the data collected was reviewed by an independent investigator to validate accuracy.

Outcomes

The primary outcome of this study was effectiveness, defined as PHQ-9 administration for patients at an outpatient encounter when the CDS tool alerted (see Table 1). Because both versions could alert more than once for the same patient and encounter, the total number of alerts for both versions of the tool were calculated based on the number of unique patient encounters. A secondary outcome was time to PHQ-9 administration using the most recent baseline PHQ-9 prior to the alert as the comparator. The time to administration was determined by taking patients with a documented PHQ-9 score on the same day of their visit and calculating the difference between that date and the date of their most recent baseline PHQ-9 score.

Statistical Analysis

Clinician response to the CDS, patient demographics, and visit characteristics were summarized descriptively. Differences in categorical demographics and visit characteristics between versions 1 and 2 were compared using chi-square tests of independence. A two-sample t test was used to test for differences in age, and Wilcoxon rank sum tests were used to evaluate differences in the distribution of number of medications between versions. Chi-square tests were used to evaluate associations between PHQ-9 administration with each version. SAS 9.4 (SAS Institute Inc) was used for analyses. The study was approved by the Colorado Multiple IRB.

Results

In February 2017, there were a total of 1987 alerts for 504 unique patient encounters with version 1 (median; IQR alerts per encounter: 3; 2, 5). In February 2018, there were a total of 917 alerts for 502 unique encounters with version 2 (median; IQR alerts per encounter: 2; 1, 2). Because version 2 was passive, the number of alerts does not reflect the number of clinicians who opted to review the alert. The mean age of patients was 51 years, and the majority were female, White, and insured. More patients exposed to version 2 had a documented diagnosis of dysthymia or depression compared with version 1 (78% vs 64%). Baseline characteristics of patients and encounters exposed to each of the tools are described in Table 2.

TABLE 2

Characteristics of patients and encounters with an alert

Characteristic	CDS Version 1, n (%)	CDS Version 2, n (%)
Age, mean (SD)	51.3 (17)	51.4 (17)
Female gender	359 (71)	376 (75)
Race
African American	100 (20)	70 (14)
White	309 (61)	359 (72)
Other	95 (19)	73 (15)
Hispanic or Latino ethnicity	82 (16)	68 (14)
Insurance
Medicare/Medicaid	256 (51)	234 (47)
Commercial	208 (41)	198 (39)
Other	40 (8)	70 (14)
Interpreter required	10 (2)	22 (4)
Diagnosis of dysthymia or chronic depression	324 (64)	391 (78)
Diagnosis of bipolar disorder	24 (5)	37 (7)
Diagnosis of personality disorder	9 (2)	15 (3)
No. of antidepressants prescribed, median (IQR)	1 (0, 1)	1 (0, 1)
No. of psychotropics prescribed, median (IQR)	0 (0, 0)	0 (0, 0)
Type of encounter
Acute	27 (5)	29 (6)
Preventative	364 (72)	466 (93)
New to establish care	113 (22)	7 (1)
Clinician type
Attending	368 (73)	443 (88)
Advanced practice provider	68 (13)	59 (12)
Resident	68 (13)	0 (0)
Type of practice
Family medicine	293 (58)	331 (55)
Internal medicine	201 (40)	159 (32)
Geriatrics	10 (2)	12 (2)
Encounter clinician is patient's PCP	241 (48)	295 (59)
Something urgent addressed at visit	21 (4)	8 (2)
Referral to psychiatrist/psychologist placed at visit	14 (2)	7 (2)
Change in antidepressant/psychotropic medication at visit	33 (7)	15 (3)
Depression/dysthymia billed at visit	2 (0.4)	2 (0.4)
Psychiatric indication billed at visit	7 (1)	0 (0)

CDS = clinical decision support; PCP = primary care provider.

Adoption and Responses to the CDS

Of the 1987 alerts with version 1, 1870 (94.1%) were not adopted (dismissed) with no documented reason, a reason was documented in 105 (5.3%) instances, and in 12 (0.6%) instances, clinicians stated they would or did complete the PHQ-9. The most common reason provided for not adopting the CDS was PHQ-9 was not appropriate (Table 3). Because version 2 was passive and did not request or allow clinicians to provide a reason for not following the recommendation, adoption rates and reasons for not adopting are not applicable.

TABLE 3

Clinician-stated responses to clinical decision support (CDS) version 1

Response	n (%)
Reason for not adopting (dismissing) provided, n = 115
Patient declined	8 (2)
Not appropriate today	68 (14)
PHQ-9 today	10 (2)
Other	29 (6)
Indicated PHQ-9 completed within CDS (yes)	2 (<1)

PHQ-9 = Patient Health Questionnaire-9.

PHQ-9 Administration Frequency and Timing

Version 2 of the tool resulted in fewer PHQ-9 administrations compared with version 1. Version 1 resulted in PHQ-9 administration 77 times (15.3% of 504 unique encounters) compared with 49 times (9.8% of 502 unique encounters) with version 2 (P = .011).

For both tools, a PHQ-9 was more likely to be administered when there was a documented antidepressant or psychotropic medication change during a given encounter (version 1: 4% with no medication change vs 22% with a change, P < .001; version 2: 2% with no medication change vs 10% with a change, P < .002). Documentation of dysthymia or chronic depression was significantly associated with PHQ-9 administration with version 2 but not version 1 (version 1: 63% documented PHQ-9 in absence of dysthymia or chronic depression vs 74% documented PHQ-9 in presence of dysthymia or chronic depression, P = .053; version 2: 76% documented PHQ-9 in absence of dysthymia or chronic depression versus 92% documented PHQ-9 in presence of dysthymia or chronic depression, P = .013). Encounters in which a referral to a psychiatrist was placed or a psychiatric indication was billed for were more likely to result in PHQ-9 administration with version 1 (version 1: 0% documented PHQ-9 in absence of referral or billing for psychiatric indication vs 6% documented PHQ-9 in presence of referral or billing for psychiatric indication, P < .001; version 2: 1% documented PHQ-9 in absence of referral or billing for psychiatric indication vs 0% documented PHQ-9 in presence of referral or billing for psychiatric indication, P = .509). Associations between encounter characteristics and PHQ-9 administration are summarized in Table 4.

TABLE 4

Associations between visit characteristics and Patient Health Questionnaire-9 (PHQ-9) administration for both versions of the clinical decision support (CDS) tool

	CDS Version 1			CDS Version 2
	PHQ-9 Not Administered, n (%)	PHQ-9 Administered, n (%)	P Value	PHQ-9 Not Administered, n (%)	PHQ-9 Administered, n (%)	P Value
Type of practice
Family medicine	238 (56)	55 (71)	.037a	295 (65)	36 (73)	.33a
Internal medicine	180 (42)	21 (27)		146 (32)	13 (27)
Geriatrics	9 (2)	1 (1)		12 (3)	0 (0)
Type of encounter
Acute	25 (6)	2 (3)	.114a	27 (6)	2 (4)	.582a
Preventative	301 (70)	63 (82)		419 (92)	47 (96)
New to establish care	101 (24)	12 (16)		7 (2)	0 (0)
Clinician type
Attending	311 (73)	57 (74)	.62a	404 (89)	39 (80)	.048a
Advanced practice provider	60 (14)	8 (10)		49 (11)	10 (20)
Resident	56 (13)	12 (16)		0 (0)	0 (0)
Encounter clinician is patient's PCP	200 (47)	41 (53)	.300	266 (59)	29 (59)	.95
Something urgent addressed at visit	18 (4)	5 (6)	.897	8 (2)	0 (0)	.348
Referral to psychiatrist placed at visit	2 (0)	5 (6)	<.001	4 (1)	0 (0)	.509
Referral to psychologist placed at visit	6 (1)	1 (1)	.941	3 (1)	0 (0)	.568
Change in medications at visit	16 (4)	17 (22)	<.001	10 (2)	5 (10)	.002
Depression/dysthymia billed at visit	0 (0)	2 (3)	.001	2 (0)	0 (0)	.641
Psychiatric indication billed at visit	4 (1)	3 (4)	.041	0 (0)	0 (0)	…
Patient has dysthymia or chronic depression	267 (63)	57 (74)	.053	346 (76)	45 (92)	.013
Patient has bipolar disorder	19 (4)	5 (6)	.438	33 (7)	4 (8)	.823
Patient has personality disorder	8 (2)	1 (1)	.7259	13 (3)	2 (4)	.6360
No. of antidepressants prescribed, median (IQR)	1 (0, 1)	1 (0, 1)	.0134	1 (0, 1)	1 (1, 2)	.0027
No. of psychotropics prescribed, median (IQR)	0 (0, 0)	0 (0, 0)	.9144	0 (0, 0)	0 (0, 0)	.8405

PCP = primary care provider.

A chi-square test was used to evaluate associations between visit characteristics and PHQ-9 administration. This statistical test evaluates whether the distribution of PHQ-9 administration across all characteristics (eg, across all 3 types of practice) differs significantly between those that were administered the PHQ-9 and those that were not.

For the secondary outcome of time to PHQ-9 administration, the median (IQR) number of days between baseline and follow-up PHQ-9 was 264 (IQR: 163, 479) days with version 1 and 229 (IQR: 144, 334) with version 2 (P = .1055).

Discussion

This study suggests CDS reminders can improve administration frequency of PHQ-9. This study also highlights how changes to specific design features can improve specificity of CDS tools with minimal impact on effectiveness. When comparing the interruptive version 1 of the tool to the noninterruptive version 2, frequency of alerts and PHQ-9 administration decreased with the noninterruptive version 2. However, the decrease in effectiveness was not in proportion to the decrease in alert frequency. Alerts decreased in frequency by more than 2-fold with version 2, yet the decrease in PHQ-9 administration was modest. Further, given that version 2 was passive, the alert frequency of version 2 overestimates the instances in which clinicians opted to view the alert. As such, the decreased effectiveness is more likely influenced by the passive, noninterruptive nature of version 2 than the decrease in alert frequency.

Others have compared active and passive tools. In general, active CDS tools are found to be more effective at changing behavior compared with passive CDS tools.8,9 Our study is consistent with these prior findings. The current study expands the body of literature evaluating the effectiveness of passive versus active CDS tools and suggests active CDS tools may be more effective but potentially at the cost of higher dissatisfaction and alert fatigue.10 Active CDS tools interrupt users' workflow at a predefined time point, which is not always aligned with the planned activity. This misalignment is a source of user frustration and can potentiate alert fatigue.

In our health system, the decrease in effectiveness with version 2 was outweighed by the decrease in alert frequency and intrusiveness; thus, version 2 has been continued. For other clinical situations of higher acuity, such a trade-off may not be optimal. Measures to decrease alert fatigue are critical in all settings, including primary care in which clinicians are flooded with alerts. Although alert fatigue was not directly measured, reducing the frequency of alerts and inappropriate alerts can improve clinician attention and response to alerts when they do appear, including those that require more urgent action.11 In our case, we added a lookback period of 18 months for the most recent PHQ-9 scores because clinician users felt this would increase the appropriateness of the CDS alerts, which also led to a substantial decrease in alert frequency.

Our findings may not be generalizable across all settings, including those with different institutional norms surrounding use of CDS tools. Health systems have different standards for the implementation and design of CDS tools as well as different capabilities based on their technical infrastructure. The findings from our study also have some limitations. As with all CDS tools, we are not able to establish causality of the change in PHQ-9 administration behavior directly with the addition of a CDS tool, but comparing the 2 tools does provide insights into the value of the different CDS designs (notably, active vs passive design). Further, we are unable to attribute changes in effectiveness of documenting PHQ-9 scores to the clinician- or medical assistant–facing CDS tools. However, our goal was to evaluate the clinician-facing CDS tool from the perspective of the impact on the clinician, including frequency of alerts and instances of adopting the CDS (did not dismiss the CDS).

Conclusion

CDS tools can improve PHQ-9 administration within primary care settings. An interruptive CDS tool may be more effective at increasing PHQ-9 administration, but a noninterruptive CDS tool may be preferred to minimize alert fatigue despite a decrease in effectiveness. The findings from this study can be considered by other institutions that are designing CDS tools to improve PHQ-9 administration and depression management.