A Qualitative Description of Clinician Free-Text Rationales Entered within Accountable Justification Interventions.

BACKGROUND: Requiring accountable justifications-visible, clinician-recorded explanations for not following a clinical decision support (CDS) alert-has been used to steer clinicians away from potentially guideline-discordant decisions. Understanding themes from justifications across clinical content areas may reveal how clinicians rationalize decisions and could help inform CDS alerts.
METHODS: We conducted a qualitative evaluation of the free-text justifications entered by primary care physicians from three pilot interventions designed to reduce opioid prescribing and, in older adults, high-risk polypharmacy and overtesting. Clinicians encountered alerts when triggering conditions were met within the chart. Clinicians were asked to change their course of action or enter a justification for the action that would be displayed in the chart. We extracted all justifications and grouped justifications with common themes. Two authors independently coded each justification and resolved differences via discussion. Three physicians used a modified Delphi technique to rate the clinical appropriateness of the justifications.
RESULTS: There were 560 justifications from 50 unique clinicians. We grouped these into three main themes used to justify an action: (1) report of a particular diagnosis or symptom (e.g., for "anxiety" or "acute pain"); (2) provision of further contextual details about the clinical case (e.g., tried and failed alternatives, short-term supply, or chronic medication); and (3) noting communication between clinician and patient (e.g., "risks and benefits discussed"). Most accountable justifications (65%) were of uncertain clinical appropriateness.
CONCLUSION: Most justifications clinicians entered across three separate clinical content areas fit within a small number of themes, and these common rationales may aid in the design of effective accountable justification interventions. Justifications varied in terms of level of clinical detail. On their own, most justifications did not clearly represent appropriate clinical decision making. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Background and Significance

Interventions informed by behavioral science embedded within the electronic health record (EHR) have been utilized in quality improvement efforts within primary care, and some have been effective. 1 2 Accountable justification alerts are one such behavioral intervention that led to a reduction in inappropriate antibiotic prescribing for acute respiratory infections. 3 4 5 Accountable justification alerts prompt a clinician who is ordering a potentially guideline-discordant test or treatment to enter a free-text rationale for why their clinical decision should override the presented clinical guideline or recommendation. Accountable justification alerts may inform the clinician that the response they enter is visible to other clinicians to provide a sense of accountability to peers. The justification exists to slow down a clinician's thinking by providing a small barrier or speed bump, prompt reconsideration, make the discordant ordering action feel non-normative, and has a social component because the justification can be seen by patients and peers. 3 When justification prompts are present, clinicians may be motivated to make better decisions, since a poor justification is likely to engender reputational concerns among peers. Several large trials are currently evaluating the use of accountable justification interventions, including improving safety of opioid prescribing, 6 reducing overtesting and overtreatment of older adults, 7 8 and, beyond primary care, for improved mechanical ventilation management. 9

Joglekar and colleagues have described a review of reasons for alert override when the justification was optional and not visible to others. 10 However, we are not aware of other studies that have reviewed visible free-text accountable justifications entered during clinical encounters in outpatient primary care to qualitatively describe what content clinicians enter.

The types of justification clinicians give provide insight into the reasons they may see as valid for guideline-discordant decisions. This information can aid in building better prompts in the future. Our objective was to qualitatively describe clinician justifications entered in the EHR when presented with an accountable justification clinical decision support (CDS) alert during routine care delivery.

Methods

Setting and Participants

We conducted a qualitative evaluation of the free-text justification rationales entered by primary care physicians participating in three separate pilot studies of CDS interventions focused on reducing potentially inappropriate opioid prescribing, high-risk polypharmacy in older adults, and overtesting in older adults. This study occurred at a large, academic regional health system in Chicago, IL. Three ambulatory primary care clinics affiliated with the health system volunteered to participate in one or more of the three pilot studies included in this analysis. The health system shares one enterprise version of an EHR (Epic, Verona, WI) and all data are copied to an enterprise data warehouse (EDW) nightly. The pilot studies ran from January to November 2019.

The Northwestern University Institutional Review Board approved both the high-risk polypharmacy study and the study on reducing overtesting of geriatric patients. The University of Southern California Institutional Review Board approved the study on reducing opioid prescribing.

Accountable justification alerts were programmed within the EHR to prompt an enrolled clinician with alerts when conditions within the chart met eligibility criteria. The pop-up alerts forced the clinician to either cancel the order or proceed with the order knowing the justification field would then be required ( Fig. 1 ). For all the three studies, if the order was not cancelled, the clinician was then required to enter text that was included in the encounter report visible to other clinicians that justified the action. If no reason was given, then the text, “No justification was given for …” was added to the encounter report. For the study aimed at reducing high-risk polypharmacy, physicians also had the option to include the justification text in the after-visit summary, a feature that was not available in the other two studies. The alerts did not otherwise restrict clinical decision making. The alerts were operationally the same in that they were programmed to fire when triggering conditions were met in the chart and that clinicians encountered a warning advisory encouraging them to cancel the order or they would see the “accountable justification alert” upon order signing. The language presented to the clinician was unique to each alert. If clinicians created any help desk tickets to internal EHR support team with complaints/concerns about study alerts, the study team would be notified.

Fig. 1

Accountable justification clinical decision support alert to reduce overtesting.

Description of Accountable Justification Interventions

Reducing High-Risk Polypharmacy

In the pilot for a high-risk polypharmacy trial, 40 clinicians (58% of the n =69 participating in the study) in two clinics received accountable justification alerts triggered by an order to renew or newly prescribe a medication meeting criteria for high-risk polypharmacy. 11 Seven different measures of high-risk polypharmacy were defined using three sources: the Beers Criteria, the Screening Tool of Older Persons' Potentially Inappropriate Prescriptions (STOPP) Criteria, and the National Action Plan for Adverse Drug Event Detection. 12 13 14 Each measure had a corresponding CDS alert for clinicians encouraging them to make safer prescribing decisions. If clinicians continued to sign a high-risk order, they received a prompt to provide a free-text rationale for the medication order. If no justification was entered, the default text “No justification was given” appeared in the justification note within the encounter report and the clinician could decide whether to share the justification in the patient's after-visit summary.

Reducing PSA Testing in Older Men and Urine Studies in Older Women with Nonspecific Symptoms

In a pilot for the BEAGLE trial, 14 clinicians (100% of participating clinicians) received accountable justification alerts if they ordered prostate-specific antigen (PSA) testing for men ≥ 76 years old with no history of prostate cancer or when they ordered a urine culture or urinalysis for women ≥ 65 years old with nonspecific symptoms. 15 The CDS alerts were triggered by guideline-discordant test ordering and encouraged the clinician to cancel the order, warning them that if they proceeded with the order they would be asked to enter a free-text rationale for the ordering decision if they proceeded with the order. If they left the field blank, “No justification was given for the order” would be added to the encounter report visible to other clinicians.

Reducing Opioid Prescribing

In a pilot for the AESOPS trial, 6 37 clinicians (90% of n =41) in three clinics received accountable justification alerts if they ordered new or renewal opioid medications for a patient. Clinicians received a pop-up CDS alert upon ordering an opioid for a patient that met triggering criteria, informing them of the risks of opioids and offering them an order set of alternative pain treatment strategies (e.g., nonsteroidal anti-inflammatory medications or physical therapy). The alert informed the clinician that if they proceeded with the opioid order, they would encounter a justification alert at order signing asking for a free-text rationale for the order. If the field was left blank, the phrase “No justification was given for prescribing an opioid” would be included under a “high-risk prescribing” section of the encounter report visible to other clinicians.

Data Collection and Analysis

For the above interventions, the accountable justification alert collected data directly within a free-text field in the alert. Each CDS element had a unique identifier. We queried the EDW to identify each instance of CDS presentation and extracted each free-text justification. We reviewed all free-text justifications entered for the three pilot studies. Some alerts triggered more often than others based on their separate triggering criteria.

Qualitative Description

The study team reviewed a random sample of 100 justifications by applying a random number to dataset and discussed primary categories of responses. Then, using a constant comparative method to iteratively develop categories, two authors (T.B. and B.Z.) coded each justification independently and then resolved all differences via discussion. 16 All justifications were given a primary code. Justifications relevant across multiple code categories were given a secondary code. Two coders reviewed all justifications once all codes were identified to ensure consistency. An “other” category was used for justifications without enough detail to code or for which no existing category was applicable.

Clinician Appropriateness Judgments

We utilized a modified Delphi approach to score justifications for clinical appropriateness. 17 Three physicians (J.A.L., T.A.R., S.D.P.) independently scored all free-text entries in the justification alert on a 1 to 9 scale. Within the scale, 1 to 3 represented likely clinically appropriate justifications, 4 to 6 represented justifications with uncertain clinical appropriateness, and 7 to 9 represented justifications that were likely clinically inappropriate. After independent scoring was completed, physician reviewers then discussed justifications where scores were more than 3 points apart or scores crossed appropriate, uncertain, or inappropriate categories. After discussion, raters were asked to reconsider their original ratings and re-rate.

Results

Accountable justification alerts triggered for n =56 unique primary care clinicians, of whom 38 (67.9%) were female. For the majority of encounters ( n =560, 64.2%), clinicians entered a free-text rationale for their ordering decision rather than opting to add the generic “no justification given” to the encounter report. Of the 560 unique justifications, 38 were related to reducing overtesting in older adults, 191 were related to reducing high-risk polypharmacy, and 333 were related to reducing opioid prescribing. Clinicians entered justifications for 42% of high-risk polypharmacy alerts, 76% of opioid prescribing alerts, and 21% of overtesting in the elderly alerts. Character length for justifications ranged from 2 to 224 with a mean of 28.7 characters (standard deviation: 24.6). There were no EHR help desk tickets created by clinicians for study alerts.

Qualitative Themes

We identified a small number of general categories or themes that described clinician justifications. We present results by three primary categories: (1) a diagnosis or symptom, (2) context of the action, and (3) special communication between clinician and patient.

Documenting a Diagnosis or Symptom

Clinicians would often enter a diagnosis or patient-reported symptom into the free-text box ( Table 1 ). When clinicians encountered alerts for potentially guideline-discordant urine testing among older women, they entered notes such as “pt has urinary frequency” and “edema.” Justifications varied in terms of clinical specificity and thoroughness. In response to request to justify PSA testing in older men, the most detailed justification was “bone pain and h/o elevated PSA without evaluation due to pt refusal in past–r/o metastatic prostate cancer.” Diagnoses for high-risk polypharmacy justifications that appeared repeatedly were anxiety, depression, pain, and insomnia/trouble sleeping. For example, justifications related to anxiety varied from brief “anxiety” to “ok for anxiety,” to more comprehensive “pt needs for inc anxiety related to breathing issues.” Not surprisingly, many justifications for opioid prescriptions focused on patient-reported pain. Some included only the painful diagnoses such as fractures or shingles and many simply entered acute or chronic pain. Other justifications included additional details such as “indicated for pain not controlled with NSAIDs for acute use” and “5 tabs for acute pain.”

Table 1

Documenting diagnosis or patient-reported symptom

Theme	Clinical behavior accountable justification alert seeks to promote	Illustrative justifications	Physician review for clinical appropriateness
Diagnosis	Reducing opioid initial prescriptions and renewals	“ kidney stone”	Uncertain
		“ inoperable chronic pancreatitis”	Appropriate
		“ Small amount for gout attack in patient with transplanted kidney”	Appropriate
	Reducing high-risk polypharmacy	“ Used for neuropathic pain from ankylosing spondylitis”	Uncertain
Symptom: pain	Reducing opioid initial prescriptions and renewals	“ Severe sciatica pain”	Uncertain
		“10/10 radicular pain, tramadol for breakthrough”	Appropriate
	Reducing high-risk polypharmacy	“ Patient has noted improved pain on gabapentin; recently required dose increase due to worsening symptoms”	Uncertain
Symptom: general	Reduce PSA screening in the elderly	“ incomplete bladder emptying, fatigue”	Appropriate
		“back pain, weight loss, rule out prostate cancer”	Appropriate
	Reduce overtesting, UA/UC	“suprapubic tenderness and urgency”	Appropriate
		“cloudy urine with odor”	Inappropriate
	Reducing high-risk polypharmacy	“burning skin sensation”	Uncertain

Abbreviations: PSA, prostate-specific antigen; UA/UC, urinary albumin to urinary creatinine ratio.

Documenting Contextual Details Defending Clinical Action

Within the broad category of providing contextual detail, there were many subcategories ( Table 2 ). Some justifications were generic, most commonly a version of the phrase “benefits outweigh risk.”

Table 2

Documenting contextual details defending clinical action

Category	Clinical behavior accountable justification alert seeks to promote	Illustrative justifications	Physician review for clinical appropriateness
Pending another approach to pain management or care	Reducing opioid initial prescriptions and renewals	“Also trying other things”	Uncertain
		“Bridging pain control until sees Spine for ESI [epidural steroid injection]”	Appropriate
		“recent fractures, weaning off while starting new pain regimen”	Uncertain
Tapering plan	Polypharmacy	“ benefits outweigh risks of prescribing medication. will work on tapering other high-risk medications and she is not actively taking many of these”	Appropriate
		“ Long history of dependence, we have already reduced doses significantly”	Appropriate
Short-term supply of medication	Reducing opioid initial prescriptions and renewals	“ using otc acetaminophen and asa. Cannot use nsaids b/c of renal disease. short course until evaluation”	Appropriate
		“short term pain control, can't take nsaids, tylenol, conservative cares not working”	Appropriate
Failed alternative treatment	Reducing opioid initial prescriptions and renewals	“ hacking cough unresponsive to other medications”	Uncertain
		“have exhausted alternatives”	Appropriate
		“ other options unsuccessful”	Appropriate
		“ pain not responsive to nsaids”	Appropriate
	Polypharmacy	“ chronic pain from OA other treatments have not worked”	Appropriate
		“ Other treatments including CBT have not helped”	Appropriate
Chronic medication	Reducing opioid initial prescriptions and renewals	“ chronic pain, on opioids for several years”	Uncertain
	Polypharmacy	“ has been on for years and stable”	Uncertain
Patient history	Reduce PSA screening in the elderly	“ past rising PSA”	Appropriate
	Reduce over-testing, UA/UC	“ history of utis”	Inappropriate
	Reducing opioid initial prescriptions and renewals	“ Pt has tolerated low doses in the past”	Uncertain
Pre/Post-op	Reduce overtesting, UA/UC	“ preop request from surgeon”	Inappropriate
	Reducing opioid initial prescriptions and renewals	“ post-op pain, weaning dose”	Uncertain

Abbreviations: PSA, prostate-specific antigen; UA/UC, urinary albumin to urinary creatinine ratio.

Among justifications entered in response to CDS to reduce high-risk polypharmacy among older adults, a common detail was mentioning chronic medication use (e.g., “Has been on chronic doses with good function and no abuse for years”).

The two most common contextual justifications for reducing opioid CDS were mentioning the prescription was for a short-term supply (e.g., “acute pain, short course” and “tolerated in past, short course”) and that the patient had tried and failed alternative treatments (e.g., “has tried 4 other therapies” and “migraine unresponsive to triptans, nsaids”).

Documenting a Discussion Between Clinician and Patient

Many justifications coded as clinician/patient discussion were detailed (“See office visit notes. Discuss risks w/ patient every visit with goal of stopping medication, however, patient has been very reluctant to do so” and “Pt has failed a number of topical therapies and other medications; she and I had a detailed discussion of r/b/se when medication was started and started the lowest dose possible. We will re-evaluate at her visit on DATE”) ( Table 3 ). However, some were as brief as “discussed.”

Table 3

Documenting details from a clinician–patient discussion

Category	Clinical behavior accountable justification alert seeks to promote	Illustrative justifications	Physician review for clinical appropriateness
Discussion: general	Polypharmacy	“ discussed possible side effects”	Uncertain
		“ pt agreeable. has care for falls”	Uncertain
	Reduce PSA screening in the elderly	“Patient's father died prostate cancer at age XX he wants continued check”	Inappropriate
		“Pt insistence”	Inappropriate
Discussion: risks/benefits	Reducing opioid initial prescriptions and renewals	“risk and benefits discussed”	Uncertain
		“ understands risk”	Uncertain
	Polypharmacy	“benefits outweigh the risks, patient understands the risks”	Appropriate
		“Patient is aware of the risk for sedation and falls with the use of sleeping medications and acknowledges the risk”	Uncertain

Abbreviation: PSA, prostate-specific antigen.

Finally, there were some justifications that were coded as “other” (e.g., “covering for Dr. X”). There were very few justifications that seemed to be written directly to the patient (“This medication can help with pain but can increase your risk of falls. Please minimize use of this medicine”).

Physician Review for Clinical Appropriateness

Given the brevity of justifications clinicians entered within the CDS alert, physician reviewers discussed n =391 (70%) justifications to agree on final scoring category and scored most justifications as “uncertain clinical appropriateness” (65%) ( Table 4 ). A larger percentage of justifications for overtesting in older adults were determined to be appropriate as compared with justifications entered for opioid prescribing.

Table 4

Clinical appropriateness of justifications by topic

Targeted clinical behavior	Total	Appropriate	Inappropriate	Uncertain appropriateness
Reduce PSA screening in the elderly	15	9 (60%)	6 (40%)	0
Reduce overtesting, urine studies	23	14 (61%)	7 (30%)	2 (9%)
Reducing initial opioid prescriptions	233	44 (19%)	7 (3%)	182 (78%)
Reducing renewal opioid prescriptions	98	27 (28%)	1 (1%)	70 (71%)
Reducing high-risk polypharmacy	191	66 (35%)	11 (6%)	114 (60%)
Total	560	167 (29%)	34 (6%)	378 (65%)

Abbreviation: PSA, prostate-specific antigen.

Discussion

This qualitative review of accountable justifications entered during routine care delivery reveals three broad themes across multiple clinical topics: (1) entering a diagnosis, (2) providing contextual detail, and (3) documenting a discussion between patient and clinician. Physician investigators found it difficult to determine the clinical appropriateness of these accountable justifications when reviewing solely the free-text data element.

These themes have some implications for accountable justification interventions. First, there is utility in reviewing free-text rationales especially at the start of using new accountable justification alerts. When a clinician gives an appropriate reason in these justifications, the CDS logic can be modified to be more clinically accurate (i.e., a diagnosis could be added to suppress an alert) and any errors in programming can be corrected. 18 This confirms the finding of Joglekar et al that suggests postimplementation evaluation of alerts, including qualitative review of free-text override responses, can contribute to the construction of better triggers that minimize unnecessary prompts that can contribute to alert fatigue. 10 Additionally, clinicians would have been spared the CDS alert in the study of overtesting of older adults if relevant diagnoses were entered into the patient problem or encounter diagnosis list because the diagnoses of a past prostate cancer history or specific urinary symptoms would have suppressed the alert. It is possible these instances will appear less often as clinicians learn from previous exposures that doing so saves them from interacting with CDS alerts. Reviewing free-text justifications will likely contribute to a more thorough understanding of clinicians' experiences with CDS prompts, which is valuable for improving them going forward, 19 20 reduction of alert fatigue, 21 and increasing clinician acceptance of these prompts. 22

Second, it may be possible to leverage these justifications to highlight common inappropriate contextual reasons or dispel a misperception either within the CDS intervention language or through another educational avenue. 23 These rationales, while brief and entered in time-constrained encounters, can provide insights into what is driving the clinical decision in real time and point to educational gaps or challenges that could be addressed. For example, the prompt for the justification initial warning alert could list the most common inappropriate rationales and flag them as poor reasons for overriding the alert or the prompt could provide a link to additional resources to help inform complex shared decision-making discussions. Ideally, the justification should include important clinical information others can use when treating the patient. While it may be unlikely a clinician will change their ordering action after encountering an accountable justification alert and the information it contains for a particular patient, it could influence their ordering decisions for future patients.

Third, it was common for clinicians to provide contextual details around their clinical decision making. Accountable justification interventions may need to highlight that while there are reasonable patient-specific reasons for guideline-discordant decisions, these deviations should be carefully considered. Treatment guidelines cannot address all the relevant details of nuanced clinical decision making. However, providing context that does not support the appropriateness of the clinical action is not useful and does not improve care. Understanding clinician perceptions and beliefs about a clinical topic through either interviews or analysis of structured and unstructured EHR data is valuable both in designing an initial accountable justification intervention 24 25 26 27 and in implementing any modifications or calibrations that are identified after an alert has been in an EHR's production environment that could enhance its effectiveness. 28

Fourth, to the extent that justification patterns differ between clinicians, with some persistently providing strong self-contained justifications for clinical decision making and others persistently providing weak or absent evidence of sound decision making, justifications such as those we analyzed could serve as the basis for developing novel performance measures for individual clinicians with attention given to minimizing the documentation burden for clinicians.

Finally, given the theme around patient/clinician discussions, mentioning this within the alert presented to the clinician may be worth exploring. Evidence-based medicine must be balanced with patient preferences through shared decision making. It may be revealed in these shared decision-making discussions that a patient's preferred action may carry significant risk 29 as is the case for the clinicians who received accountable justification alerts for these studies. Calling attention to the downside of the patient's preferred action may be an opportunity for clinicians to change a patient's mind about a test or treatment after alert exposure. We acknowledge these discussions, particularly around deprescribing, 30 31 are challenging and behavior change often requires interventions aimed at multiple functions. 32

Limitations

This study has several limitations, including a small sample of clinicians within a single health system. Additionally, given we reviewed only the accountable justification data element entered by the clinician during a visit and not the entirety of the encounter report, it is not surprising that many of the clinician appropriateness scores were judged as “uncertain.” We do not expect this to be a method to fully capture the nuanced decision making that takes place between a patient and their clinician. Lastly, this approach to informing the development of future decision support provides no insight into alerts that should have triggered but did not (false-negatives).

Conclusion

Our qualitative description of justifications entered by clinicians when ordering potentially guideline-discordant tests and treatments highlights three themes of entering a diagnosis, providing contextual details, or noting a discussion between clinician and patient. Qualitatively reviewing free-text justifications has utility at the beginning of an intervention period to fine-tune CDS logic and may highlight common clinician misconceptions that could be addressed to improve the effectiveness of accountable justification interventions and improve clinical care. Future studies should explore whether these justifications can be safely shared with patients, whether doing so enhances the value of the justification interventions, and whether justifications can serve as a new basis for measuring clinician performance.

Clinical Relevance Statement

Qualitatively evaluating CDS interventions that request a free-text justification to proceed with potentially guideline-discordant care may improve the overall effectiveness of CDS alerts in improving quality of care.

Common justifications provided by clinicians when prompted with accountable justification alerts were:

Entering a diagnosis

Providing contextual details

Documenting a discussion between patient and clinician

All of the above

Correct Answer: The correct answer is option d. We found that clinician justifications fell into three main themes of entering a diagnosis, providing contextual details, or documenting a discussion between patient and clinician.

Reviewing free-text justifications entered by clinicians within clinical decision support alerts is useful only at the conclusion of a study.

True

False

Correct Answer: The correct answer is option b. Reviewing free-text justifications has utility across the implementation cycle of accountable justifications within the EHR and may help identify ways to improve the effectiveness of these interventions.