Implementing evidence-based anticoagulant prescribing: User-centered design findings and recommendations.

Background: Direct oral anticoagulants (DOACs) are widely used medications with an unacceptably high rate of prescription errors and are a leading cause of adverse drug events. Clinical decision support, including medication alerts, can be an effective implementation strategy to reduce prescription errors, but quality is often inconsistent. User-centered design (UCD) approaches can improve the effectiveness of alerts.
Objectives: To design effective DOAC prescription alerts through UCD and develop a set of generalizable design recommendations.
Methods: This study used an iterative UCD process with practicing clinicians. In three rapid iterative design and assessment stages, prototype alert designs were created and refined using a test electronic health record (EHR) environment and simulated patients. We identified key emergent themes across all user observations and interviews. The themes and final designs were used to derive a set of design guidelines.
Results: Our UCD sample comprised 13 prescribers, including advanced practice providers, physicians in training, primary care physicians, and cardiologists. The resulting alert designs embody our design recommendations, which include establishing intended indication, clarifying dosing by renal function, tailoring alert language in drug interactions, facilitating trust in alerts, and minimizing interaction overhead. Conclusions: Through a robust UCD process, we have identified key recommendations for implementing medication alerts aimed at improving evidence-based DOAC prescribing. These recommendations may be applicable to the implementation of DOAC alerts in any EHR systems.

Improper prescribing of direct oral anticoagulants is a leading cause of adverse drug events.

Electronic health records can use alerts to reduce errors, but the alerts must be designed well.

We produced easy‐to‐use medication alerts through a user‐centered design process.

The principles for designing effective alerts could help others design more effective alerts.

INTRODUCTION

Direct oral anticoagulants (DOACs) have revolutionized chronic anticoagulation therapy practices since their introduction in 2008 and are now first‐line therapy in common thrombotic conditions. Despite a more than 10‐year adoption period, unacceptable rates of prescribing outside of the evidence‐based recommendations persist (“prescribing errors”). These errors are not without consequence. DOAC medications are one of the leading causes of adverse drug events, and inappropriate prescribing of DOACs is associated with higher rates of bleeding and thrombotic events. These prescribing errors often arise due to unique dosing based on the specific DOAC medication, clinical indication, time since diagnosis, renal and/or liver function, and drug–drug interactions.

While often employed to reduce inappropriate prescribing, medication alerts built into the electronic health record (EHR) have produced inconsistent results. , , A review of clinical decision support (CDS) studies concluded that the varying quality of CDS tools may account for the mixed results, limiting the ability to universally recommend CDS. Importantly, the quality of the design and implementation of alerts is a key factor in their effectiveness. User‐centered design (UCD) is a participatory design process involving users and designers working together to iteratively develop and refine a product or interface. Employing a UCD process for EHR alert development has been shown to improve alert effectiveness in numerous contexts. , ,

With the aim of reducing prescribing errors, we used a UCD process involving interviews with prescribers to develop DOAC medication alerts for an EHR. The goal of this study was to design a set of user‐friendly alerts, along with guiding principles, that could be implemented within and extended to any health system or EHR, with the aim of improving evidence‐based DOAC prescribing given known barriers of alert fatigue and communication overload.

METHODS

This study employed an iterative UCD process to develop two different DOAC medication alerts. The first alert is intended to be displayed at the time a prescription is written to identify a potentially incorrect DOAC prescription before it is sent to the pharmacy. The second alert is intended to be delivered to the prescriber in the days, weeks, or months after a prescription has been written, when a change in a patient's clinical situation makes his or her currently used DOAC dose inappropriate (e.g., change in renal function, new drug–drug interaction).

Participants in this study were practicing clinicians with medication‐prescribing authority. These participants were selected in a stratified sample to vary in their levels of experience (attending physicians, resident physicians in training, advanced practice providers) and expertise (generalists, specialists). Participants were recruited through email (GB), and meetings were held via private video link (due to COVID‐19 pandemic restrictions). The study was conducted using a combination of an EHR “test environment” and interactive design prototypes that presented realistic (but fabricated) information about a patient being prescribed a DOAC. The protocol was reviewed by the Institutional Review Board at the University of Michigan, deemed minimal risk, and exempt from ongoing oversight.

We used an ethnographic‐enhanced user‐centered iterative design approach to develop both CDS tools. , , This process involved rapid iterations of prototype designs for the system's user interface and user experience. This approach is rooted in perspectives of design thinking, design research, and anthropological research.

Following recognized best practices in UCD, the software design and development process progressed through three main stages: (i) user discovery to identify and prioritize user needs, (ii) hypothesis‐driven rapid iterative design and assessments to evaluate appropriate messaging and layout, and (iii) hypothesis‐driven rapid iterative design and assessments to validate the optimal user experience given potential technical constraints.

User‐centered design process

A combination of test systems and interactive prototypes were used to collect data in three stages (Table 1). The information from each stage was synthesized, and the insights drove subsequent design revisions to arrive at a proposed validated solution.

TABLE 1

Overview of the user‐centered design process

Stage	Activity	Goal	Participants
Stage 1: User discovery	Testing initial design Cognitive walk‐throughs (four scenarios)	Understanding general prescribing behavior	Advanced practice providers Primary care physicians Specialists (MD)
Stage 2: Rapid iterative design and assessment	Translating insights from Stage 1 into refined designs (four scenarios)	Refining and testing initial DOAC‐specific designs	Trainees (MD) Primary care physicians Specialists (MD)
Stage 3: Rapid iterative design an assessment	Validating refined designs (five scenarios)	Validation of solutions and integration into workflow	Advanced practice providers Primary care physicians Specialists (MD)

Abbreviation: DOAC, direct oral anticoagulant.

We began the user discovery stage (Stage 1) by testing an initial set of hypothetical designs crafted by the clinical subject matter experts. The goal at this stage was to document our understanding of general prescribing behavior, overall knowledge of DOAC‐related contraindications, and response to alerts. To gather this information, we conducted design assessments in which we presented a scenario to a user; we then observed as the user responded to the scenario through the proposed alert interface, and then discussed the proposed alert interfaces in semistructured interviews led by professional UCD facilitators (MC, MP).

In the rapid iterative design and assessments stage (Stage 2), we translated Stage 1 insights into a revised set of contrasting design solutions for the user interface. The goal at this stage was to uncover the best way to present critical DOAC medication alert and warning information in a way that allowed clinicians to evaluate the content of the message appropriately, while minimizing unnecessary interruptions and gaining clinician trust. To gather this information, we conducted four scenario‐driven facilitated design assessments similar in structure to Stage 1 assessments.

Stage 3 involved additional rapid iterative design and assessments, wherein we translated Stage 2 insights into successively refined revised solutions for the user interface and experience. The goal at this stage was to validate a potential solution for presenting DOAC medication alert and warning information in an idealized workflow given the potential constraints of the EHR system.

Across these three stages, we assessed the designs with representative clinicians who prescribe anticoagulant medications in different settings (e.g., emergency room, primary care, and cardiology providers). All sessions with users began with an explanation of the goals of the research. Additionally, we compiled field notes from the direct observations and recorded all interviews. We reviewed the field notes and recorded interviews to identify key emergent themes across all users, reconciling our understanding of this field data between the designers (MP, MC) and subject matter experts (FJS, ML, GDB). Our analysis was grounded in both the theoretical perspectives of situated action and embodied cognition as well as medical anthropology traditions.

Qualitative data analysis

Data collection included qualitative assessments of user success, failure, satisfaction, and frustration in response to the prototypes and assessment scenarios. Key themes and key discrepancies across participants were documented in each round of assessments and informed subsequent design revisions. These themes drove each subsequent round of rapid iterative design until there was strong alignment (saturation) among participating clinicians around the Five Rights of CDS design: the right information, to the right person, in the right format, through the right channel, at the right time. For each stage, we recruited and interviewed participants until thematic saturation was reached, that is, when additional participants sessions were not unveiling meaningfully new information.

RESULTS

A total of 13 providers comprised our UCD sample, including two advance practice providers (one nurse practitioner, one physician assistant), two physicians in training, six attending primary care physicians, and three attending cardiologists (Table 2).

TABLE 2

Participants in user‐centered design sessions

	Advanced practice providers	Physicians in training	Primary care physicians	Cardiologists	Total
Stage 1	1	0	2	1	4
Stage 2	0	2	1	1	4
Stage 3	1	0	3	1	5
Total	2	2	6	3	13

Early on, answers to three key questions emerged as critical to successful messaging: (i) Why am I seeing this alert or message? (ii) Does this risk really apply to my patient? and (iii) What should I do next? To address these critical questions, five key design themes emerged from the UCD process that aim to improve correct DOAC prescribing within the EHR (Table 3).

TABLE 3

Design principles and examples

Design principle	Example
Establish intended indication	Prompt user when indication is missing An indication is required for medication/dose selection logic Prompt is efficient for workflow
Clarify dosing by renal function	Cite Cockcroft–Gault equation to highlight renal function (creatinine clearance) Contrast with glomerular filtration rate
Tailor alert language in drug interactions	Name both drugs Be specific, concrete, and brief Provide alternatives
Facilitate trust in alerts	Name and include reference source Use clinic/individual “letterhead” Include only alerts viewed as “valid”
Minimize interaction overhead	Constrain responses Predict information needs

Establish intended indication

Appropriate DOAC dosing varies depending on the patient's indication. For example, apixaban 2.5 mg twice daily is an appropriate apixaban dose for patients with an indication of thromboprophylaxis after VTE, but is inappropriate for a patient with atrial fibrillation who is younger than 80 years of age with normal renal function. Associating an indication in the patient's record is therefore necessary to determine whether a given DOAC prescription is appropriate. Two design recommendations follow. First, it is recommended that if there is no indication on record when prescribing a DOAC, an alert should prompt a required response for indication before allowing the prescriber to complete the prescription. A second, contingent requirement is that any prompt of the prescriber to input an indication should involve minimum manual input, ideally involving only a single click.

Clarify dosing by renal function

Renal function is commonly assessed by calculating creatinine clearance (CrCl) or estimated glomerular filtration rate (eGFR) using one of several different equations. These two measures are not interchangeable and may have important implications for evidence‐based DOAC dosing. , ,

Several design implications stem from the need to use CrCl instead of eGFR. Primarily, estimates of renal function within the alert should display only the CrCl since eGFR is not relevant in this context but often relied on by clinicians who were less familiar with evidence‐based DOAC dosing. An alert dialog should also not provide eGFR calculations in displayed lab values, as this can introduce confusion for the prescribing clinician. Furthermore, when providing information on renal function within the alert, clearly show which equation was used and the associated lab values. A brief comment explaining why the eGFR is not used for DOAC dosing provides further guidance for the prescribing clinician.

Finally, renal function notoriously fluctuates. As such, the prescribing clinicians strongly preferred to see the trend in renal function over time. This was particularly relevant for patients with CrCl near the cut points for dosing changes. Clinicians indicated that this trend information allowed them to determine the most clinically appropriate response.

Tailor alert language in drug interactions

Alerting prescribers of drug–drug interactions is important when using DOACs. Medication alerts for these interactions are currently available in many EHR systems. However, elements of content and language used in current drug–drug interaction alerts vary widely, and many users found the language confusing and/or the content unhelpful. Our final design reflects the following recommendations from prescriber feedback.

Name both drugs when identifying the problem

When citing a drug interaction, it was important to cite both the DOAC and the other interacting drug by name to ease the mental burden while trying to answer the questions “Why am I seeing this alert or message?” and “Does this apply to my patient?” Naming the class of drug (e.g., cytochrome P450 [CYP] 3A4 inhibitor) was not sufficient in prescriber‐facing alerts.

Be specific, concrete, and brief to maximize efficiency

Specific language should name the drug, as opposed to the class of drugs, and name the specific effect of the interaction as opposed to the class of interaction. Concrete language should name the outcome of an interaction in terms of concrete health implications (e.g., increase bleeding risk) as opposed to physiologic mechanisms (e.g., is an inhibitor). Brevity was important to increase the chance that clinicians would read and act on the content of the alert. Meeting the requirement for brevity precluded an “inclusive” alert that named both general and specific aspects of the drug–drug interaction. As a result, we recommend language similar to the following in describing the interaction between rivaroxaban and ritonavir:

“Concurrent use of rivaroxaban and ritonavir is CONTRAINDICATED. It can lead to bleeding risk from elevated rivaroxaban levels.”

More detailed language used in our earlier designs, while more informative, was not well received by the prescribing clinicians in our study. Therefore, we would recommend avoiding language such as:

“Concurrent use of rivaroxaban and combined CYP3A4/P‐gp inhibitors can increase rivaroxaban bioavailability, leading to unsafe elevated rivaroxaban serum levels.”

Clinicians wanted explanations of the interaction mechanisms to be available in a separate section of the alert.

Provide relevant actions, including DOAC and alternative medications

When the prescribed DOAC interacts with other medication, explicitly acknowledge whether an alternative DOAC would be an acceptable substitute. Prescribing clinicians in our study often wondered if an alternative DOAC would interact similarly when presented with a drug–drug interaction warning. These clinicians acknowledged that the time required to answer questions about the appropriateness of alternative drugs would be a significant burden and barrier if not presented in the CDS itself. Additionally, providing options for easily asking questions within the system and/or transferring follow‐up responsibility to an appropriate resource (e.g., DOAC specialist, anticoagulation clinic pharmacist) was important for directing appropriate action.

Facilitate trust in alerts

A prescriber faced with an alert must assess the validity and relevance of the information presented. Our design recommendations include techniques to increase users' trust in an alert message.

We facilitated prescribers' assessment of alert validity in two ways: We provided equations used for calculating CrCl, and we showed the parameters that are used in dosing calculations.

We facilitated trust in three ways in our alert designs. First, when recommending a given action, we named and included links to official institutional policies, peer‐reviewed publications, and national guidelines that could be viewed directly to support the recommendation. The presence of trusted sources was key to building credibility in alert messages, even if they were not directly referenced by clinicians.

Second, clinicians responded more favorably to messages originating from a trusted entity such as an anticoagulation clinic, anticoagulation team, or known institutional specialist/attending/clinic director. In our designs, this was accomplished by using clinic or individual “letterhead” in messages and including a salutation and signature from a trusted individual. When sending a recommendation to change a prescription, users in our study expressed distrust of generic system messages.

Third, we engendered trust by “being trustworthy.” Alerts that contain recommendations that are not supported by strong evidence, are controversial, or contextually irrelevant (nuisance alerts) are likely to be ignored. Our design recommendation is to maximize responses to alerts by following the maxim: Less is more. Include only those alerts that will be widely viewed as valid by most users in most cases. In the case of evidence‐based DOAC prescribing, we chose to exclude alerts pertaining to liver function while including alerts on renal function. In contrast to the solid evidence and applicability of renal function dosing within the EHR, liver function measurement and classification is less precise when attempting to use only discrete EHR data. As such, clear DOAC dosing/use guidance based on liver function cannot be automatically provided without the risk of frequent false‐positive alerts.

Minimize interaction overhead (fewer clicks)

A universal axiom of interface design is to design efficient interfaces. In DOAC alert design recommendations, this axiom is expressed by designing alerts to provide clear and concise responses to a suggested dose or drug change with the fewest clicks. This can be achieved by following these specific recommendations.

Constrain responses

Most actions should be completed with a single click (e.g., choosing one of four possible alternative medications in response to incorrect dose alert), as opposed to requiring prescriber to input drug, dose, and frequency manually. For DOACs, it is possible to generate a small number of selectable options for rivaroxaban or apixaban. Use buttons in the interface when possible, and if buttons are not practical, then use constrained options for the most common options (e.g., 2.5, 5, and 10 mg). Avoid using fillable numeric fields when possible. This increases speed and reduces manual transcription errors.

Provide structure

If typing is necessary, then support user input when possible. For example, provide templates for responses and pull data from the chart as possible to populate the template.

Predict information needs

The most relevant information should be directly visible, and not require a click to see. Users commented that seeing trends in CrCl was useful and they preferred to see only lab values relevant to the given alert.

These design recommendations are reflected in the design prototype (Figure 1).

FIGURE 1

Example of alert design embodying the design recommendations by (1) naming both drugs (for drug–drug interaction alerts); (2) using specific, concrete, and brief language in message; (3) providing definitive reference sources; (4) predicting information needs by providing relevant labs; (5) providing a constrained set of responses requiring few clicks; and (6) including alternative drug recommendations to prescribe. CrCl, creatinine clearance; CYP, cytochrome P450; P‐gp, P‐glycoprotein

DISCUSSION

DOAC prescribing errors are a prevalent and impactful problem to be addressed. Alerts and notifications can be effective implementation strategies to reduce medication errors and adverse events but only if they are well designed. Our domain‐specific findings are the results of resource‐intensive UCD efforts beyond the means of most health care systems. Our design recommendations may help other health care systems implement high‐quality alerts and notifications. The DOAC‐specific recommendations may also be generalized to other alerts, especially other medication alerts.

To maximize the impact of the clinical decision‐support process, we recommend that providers be prompted to input a diagnosis or indication before completing a prescription when one cannot be identified within the medical record (e.g., on the problem list). Using a “hard stop” is not often recommended in interfaces and can lead to user frustration. However, it is justified in this case since the prompt should be specific to the clinical situation, will meaningfully impact medication prescribing recommendations, and will ultimately improve workflow. Test users reported that if not for the diagnosis/indication prompt, they would be forced later to reopen the patient's chart in a much more cumbersome, time‐consuming process. This hard stop improves both safety and efficiency.

Clarifying the DOAC dose by renal function embodies a more general principle of anticipating user needs, providing information in the form it is needed, and providing “system transparency.” Together, this approach helps users understand how a system is performing calculations and why a specific recommendation is being made. Estimating renal function is a critical step in prescribing DOACs, influencing both dose selection and assessment of DOAC appropriateness. Providing the equation used for renal function calculations as well as the laboratory values used in those equations increases the transparency of the system and bolsters trust in the calculated value. Automatically computing and presenting the renal function calculations saves users' effort and reduces the chance of errors.

The design issue surrounding CrCl versus eGFR brings about an additional opportunity for “just‐in‐time” education specific to DOAC prescribing—namely, that CrCl and not eGFR is used to dose DOACs. This is particularly relevant given that most EHRs default to showing the eGFR, which can diverge significantly from CrCl within the range that DOAC dosing changes are often impacted (i.e., 30–60 ml/min). This approach demonstrates how design can be used to improve workflow, increase safety, and provide just‐in‐time education in a targeted and relevant manner.

Our design recommendations also demonstrate the principle of designing for “mental economy.” The goal of mental economy is to anticipate user needs and provide the necessary information in a form that requires minimum mental effort and transformation before application when making a critical decision. In our recommendation to name specific, concrete outcomes of a drug–drug interaction (“bleeding risk from elevated rivaroxaban levels”) instead of abstract effects and mechanisms (inhibitor of CYP3A4 leading to increased bioavailability), we anticipated that the latter message would require the user to reason through the implications of inhibition and bioavailability. Providing the outcome of that reasoning saves mental effort of the user.

Mental economy also applies to designing the possible responses to an alert. We recommend providing a small number of discrete choices for “correcting” the DOAC order rather than asking prescribers to generate their own new medication order. Psychophysical principles of decision making include well‐established equations dictating the increased time needed to process each additional option presented. , Behavioral sciences and usability testing have long established that it is easier to recognize a correct option as compared to remembering what options are available. When designing alerts, reducing the number of options presented allows users to recognize the proper response instead of having to recall it.

Our design recommendations are intended to bolster trust in the alerts. This is achieved by increasing the predictive validity of alerts (fewer nuisance alerts), providing trusted sources for our decision support, and providing system transparency. These approaches apply broadly to alerts across domains, and considerable work has been published on the topic of trust in automation in high‐risk domains such as aviation and nuclear power. The same principles can be applied within health care, beyond DOAC medication alerts.

Limitations

While our UCD approach allowed for rich data to inform these recommendations, certain limitations must be acknowledged. First, these recommendations were generated from a process focused on appropriate DOAC prescribing and may not apply in totality to other medications. Second, the participating prescribers were a limited number of users sampled from a single, integrated health system and may not represent the preferences and behaviors of prescribers from other health systems or other countries. Finally, the alerts were designed loosely around the Epic EHR system and may not directly translate to other EHR systems. Nonetheless, the use of robust qualitative and UCD methods should help to increase the applicability of the key findings and recommendations for medication alert development.

CONCLUSIONS

DOACs improve outcomes for patients with thrombotic conditions but frequently lead to harm when prescribed inappropriately. Through a robust UCD process, we have identified key recommendations for implementing medication alerts aimed at improving evidence‐based DOAC prescribing. As it is not practical for every alert to be developed using a UCD process, the design recommendations discussed here can be applied to most medication alerts. Furthermore, the specific techniques of system transparency, mental economy, and engendering trust are examples of global strategies that can be effective more generally for the design of any alert or notification system. Applying these recommendations to the implementation of DOAC alerts likely faces few technical barriers, as incorporating the recommendations should not require particularly novel or sophisticated programming.

AUTHOR CONTRIBUTIONS

G. Barnes, M. Lanham, M. Pomorski, and M. Callahan all contributed to the concept, design, analysis and interpretation of the data. F.J. Seagull contributed to the analysis and interpretation of the data, critical writing, and revising of the intellectual content. E. Jones contributed to data collection, interpretation of data, and critical revisions of the manuscript. All authors contributed to critical revising of the intellectual content, and final approval of the version to be published.

RELATIONSHIP DISCLOSURE

G. Barnes has received consulting fees from Pfizer, Bristol‐Myers Squibb, Janssen, and Acelis Connected Health, and served on the board of AC Forum. No other authors report conflict of interest.