A novel strategy for evaluating the effects of an electronic test ordering alert message: Optimizing cardiac marker use.

BACKGROUND: Laboratory ordering functions within computerized provider order entry (CPOE) systems typically support the display of electronic alert messages to improve test utilization or implement new ordering policies. However, alert strategies have been shown to vary considerably in their success and the characteristics contributing to an alert's success are poorly understood. Improved methodologies are needed to evaluate alerts and their mechanisms of action.
MATERIALS AND METHODS: Clinicians order inpatient and emergency department laboratory tests using our institutional CPOE system. We analyzed user interaction data captured by our CPOE system to evaluate how clinicians responded to an alert. We evaluated an alert designed to implement an institutional policy restricting the indications for ordering creatine kinase-MB (CKMB).
RESULTS: Within 2 months of alert implementation, CKMB-associated searches declined by 79% with a corresponding decline in CKMB orders. Furthermore, while prior to alert implementation, clinicians searching for CKMB ultimately ordered this test 99% of the time, following implementation, only 60% of CKMB searches ultimately led to CKMB test orders. This difference presumably represents clinicians who reconsidered the need for CKMB in response to the alert, demonstrating the alert's just-in-time advisory capability. In addition, as clinicians repeatedly viewed the alert, there was a "dose-dependant" decrease in the fraction of searches without orders. This presumably reflects the alerting strategy's long-term educational component, as clinicians aware of the new policy will not search for CKMB when not indicated.
CONCLUSIONS: Our analytic approach provides insight into the mechanism of a CPOE alert and demonstrates that alerts may act through a combination of just-in-time advice and longer term education. Use of this approach when implementing alerts may prove useful to improve the success of a given alerting strategy.

INTRODUCTION

Optimal patient care requires clinicians to modify their laboratory test ordering practices in response to advances in medical knowledge and technology. However, despite convincing evidence or guideline changes, some clinicians may continue ordering tests according to historic practices. Among the resources commonly leveraged by institutions to influence test ordering practices is the computerized provider order entry (CPOE) system.[1-5]

CPOE systems allow clinicians to directly input orders for diagnostic and therapeutic measures into computers. CPOE systems often include a laboratory test ordering module, which may be configured to permit the display of test-specific ordering alert messages or alerts.[1] Ordering alert messages may contain information such as testing indications, alternative tests to consider, cost, turnaround time, and assay limitations.[15-7] Sophisticated systems can offer more advanced decision support by customizing alerts to remind clinicians of patient-specific issues such as redundant test orders or suggested corollary orders.[1-38-13] Further, alerts can be noninterruptive or interruptive.[112] Noninterruptive alerts are passively displayed during the ordering process and do not require action on the part of the ordering provider. Interruptive alerts stop the workflow, requiring the provider to acknowledge the alert, and may additionally require a specific action such as entering an indication for testing.

Laboratories and institutions may design alerts to change test ordering practices in response to inappropriate test utilization or to implement new testing guidelines.[1-314] One of the likely benefits of such alerts is that they provide just-in-time advice. As clinicians have limited time to reference textbooks, medical literature, and other traditional sources of knowledge during routine patient care, provision of just-in-time knowledge has been suggested as a strategy with great potential for improving clinical decision making.[415] Alerts, in addition to their immediate effects on workflow and decision making, may also provide a longer term educational benefit with utility beyond the current ordering session. Despite the potential of a well-designed alert strategy, ordering alerts have been shown to vary considerably in efficacy[314] and the precise characteristics and mechanisms contributing to an alert's success are not well understood.

In this report, we perform a detailed analysis of an alert used to implement an institutional decision to change the acceptable indications for creatine kinase-MB (CKMB) testing. Based on recent clinical guidelines,[16] our institution restricted CKMB testing for all indications except postpercutaneous coronary intervention (post-PCI). This represented a significant change, as CKMB was previously used in combination with troponin to diagnose or exclude routine acute myocardial infarction, assess infarct size, and monitor reinfarction. The new policy stated that troponin measurement alone should be used for these purposes. The challenge in substantially changing CKMB indications without banning the test altogether necessitated a well-designed CPOE alert strategy. We developed a novel alert evaluation methodology and applied it to the alert for CKMB. In creating this methodology, our aim was to develop a framework to better study and quantify the mechanisms and effects of a CPOE alert with the ultimate goal of improving alert design.

MATERIALS AND METHODS

Setting

The Massachusetts General Hospital (MGH) is an 898-bed tertiary care academic hospital in Boston, MA. The MGH Clinical Laboratories in the Department of Pathology support the inpatient medical, surgical, pediatric, and obstetric services of the hospital, as well as primary care and specialty outpatient practices in the greater Boston area. The clinical laboratories include core laboratory (chemistry, hematology, and immunology), microbiology, blood transfusion services, and various specialty laboratories. This study was conducted with approval of our hospital's Institutional Review Board (IRB).

Provider Order Entry System and Knowledge Management Middleware

The MGH Computerized Provider Order Entry system (CPOE) is an internally developed application. The CPOE system is used by inpatient and emergency room providers to place all patient orders including laboratory tests. The primary users of our CPOE system are residents in training, clinical fellows, attending clinicians, medical students, nurse practitioners, and physician assistants. Trainees comprise the largest group of ordering providers, accounting for over 90% of CPOE orders.

As described previously, the laboratory order entry module of the CPOE system utilizes a search methodology that accepts a search string from the user and returns a list of tests matching the search string.[17] The system maintains a list of search terms (test synonyms, common misspellings, and associated clinical conditions) used to match available tests to a given search string. Users select desired tests from the search results. Representative screen shots from the CPOE system are shown in Figures 1a and b. The system tracks user interactions including searches entered and tests ordered. Each event is time stamped and includes information about the patient, provider, search terms used, and tests ordered.

Figure 1

Screenshots of the laboratory test search module are shown. Note the ordering message (noninterruptive alert) displayed for CK isoenzymes (CKMB + CPK) shown in (a). Clinicians selecting CK isoenzymes from the search screen (by clicking “Add”), encounter the interruptive alert screen (b). The interruptive alert requires that the user enter an indication for testing or cancel the testing

As previously described, a novel middleware system interacts with the CPOE system and is used to provide real-time updates to test ordering messages, search terms and other test parameters.[17] The middleware can also be used to design interruptive alert windows to be displayed during the test ordering process and request CPOE user information regarding the test. Representative screen images of the middleware are shown in Figures 2a and b.

Figure 2

(a) A representative screen shot of the middleware knowledge management system. Ordering messages (noninterruptive alerts) can be added in a matter of minutes by changing the “SQL Additional Message” field and transmitting the update to the live system. Interruptive alerts can be built with using interface shown in (b)

CKMB Ordering Guidelines Changes

Effective 3/21/11, hospital guidelines were changed to restrict the ordering of CKMB solely to patients following percutaneous coronary intervention. This change was made in consultation with representative clinical staff including cardiology, medicine, and emergency medicine. In support of this change, a noninterruptive ordering message describing the new policy was added to the CKMB search results screen [Figure 1a]. Clinicians continuing past this noninterruptive message (by clicking “Add” to order CKMB) encounter an interruptive CPOE alert Figure 1b, requiring them to withdraw the CKMB order or enter an indication for testing. The guideline change was also communicated through several other means including email announcements.

Collection of CPOE User Interaction Data

Data for the study were compiled from CPOE searches and orders from February 1, 2011 to June 1, 2011. CKMB orders placed using free text, add-on test orders, or templates were not included in our analysis since these would not trigger the alert. CKMB orders placed using free text and add-on testing accounted for less than 1% of total orders. In addition, template orders, which accounted for less than 10% of total CKMB orders both pre- and postintervention, were presumably appropriate, as templates were updated to conform to the policy change.

Analysis of CPOE User Interaction Data

The captured data did not directly link searches and orders. To retrospectively link searches and orders, an order was deemed associated with a search if the search and order were for the same patient and provider and the order followed the search within 72 minutes (one twentieth of a day). While most often searches and associated orders occurred within a much shorter period, we used 72 minutes to account for the likelihood that providers may occasionally be interrupted during the ordering process.

We analyzed user data to compile a list of terms used to search for and order CKMB. Common CKMB-associated search terms included “cardiac markers,” “CKMB,” “MB” and “ck”. In order to assess the user response to the alert we needed to determine the probable user intent when users searched using a CKMB-associated search term, as many search terms could be used to order CKMB or alternative tests. For example, searching for "creatine kinase" would result in both CKMB and CPK being retrieved and therefore the clinician could order either CPK or CKMB. In determining whether a search using a CKMB-associated term was intended to create a CKMB order, we applied the following criteria:

If the search was associated with an order for CKMB, it was presumed to be intended for CKMB and classified as “CKMB order generating.”

If the search was only associated with orders for alternative tests retrievable using the search term, it was presumed to be intended for an alternative test and classified as “alternative test order generating.”

If a search using a CKMB-associated term was associated with neither a CKMB order nor an order for an alternative test the search was classified as “nonorder generating.” To determine baseline provider intentions when using each term and thus how to count nonorder-generating searches, we examined data from the preintervention period (February 2011). We calculated a ratio (W) for each search term according to the following formula: Wi=Ci÷(Ci+Ai) where Wi equals the weighting factor for term i, Ci equals the number of searches in the preintervention period using term i that were CKMB-order-generating (criterion a, see above) and Ai equals the number of searches in the preintervention period using term i that were alternative test order generating (criterion b). A list of search terms used to order CKMB, weighting factors and associated data is shown in Table 1.

Table 1

Weighting factors used in analysis

In the tabulation of searches using CKMB-associated search terms across the entire study period, CKMB-order-generating searches (criterion a) were counted as one search. Likewise, alternative-test-order-generating searches (criterion b) were counted as zero CKMB searches. Finally, nonorder-generating searches (criterion c) were counted as Wi fraction of a search. Except as otherwise noted, data were compiled and analyzed using Microsoft Access 2003 and Microsoft Excel 2003, including built-in statistical functions.

RESULTS

CKMB orders decreased by approximately 80% within weeks of alert implementation [Figure 3]. The total number of searches using CKMB-associated search terms decreased 79%, from 1,120 per month in the preintervention period (February 2011) to 233 per month within 2 months following implementation (P<0.01). Order-associated searches (searches for CKMB followed by an order for CKMB) decreased 87%, from 1,106 to 139 per month (P<0.01).

Figure 3

Searches using CKMB-associated search terms. Shown are 7-day rolling averages for CKMB searches per day, CKMB searches associated with CKMB orders, and the proportion of CKMB searches not associated with CKMB orders with the values for the dotted and dashed lines on the left vertical axis and the solid line on the right vertical axis. The alert was implemented on March 21, 2011 as indicated by the black triangle

Further, as shown in Figure 3, the proportion of searches not resulting in orders markedly increased following the introduction of the alert, from 1% in the preintervention period (February 2011) to 40% postintervention (April 2011, P < 0.01). Since clinicians searching for CKMB and then deciding not to order the test in response to an alert would log a search without an associated order, the proportion of searches not resulting in orders provides an indication of the just-in-time advisory effect of the alert. To confirm of our strategy of weighting nonorder-generating (ambiguous) search terms, we performed a second analysis of search data, analogous to Figure 3, but confining searches to the unambiguous terms “CK MB,” “CKMB ,” “CK-MB,” “CKM,” and “MB.” The results of this analysis including only unambiguous terms are consistent with the results of our primary analysis (data not shown).

To investigate the long-term educational component of the alert, we hypothesized that the proportion of searches not resulting in orders would decrease as clinicians are increasingly exposed to the alert. Thus, clinicians who had previously seen the alert would be more likely to be aware of the new guidelines and would be less likely in the future to search for CKMB when not indicated. Figure 4 shows the proportion of CKMB-associated searches not resulting in orders as a function of the number of times that provider had previously searched using a CKMB-associated search term and thus likely viewed the alert. As shown, in a “dose-dependant” fashion, providers who had not previously seen the alert (i.e., 0 prior searches) were more likely to place a search without an associated order compared to providers that had seen the alert one or multiple times (P<0.01).

Figure 4

Analysis of provider actions grouped by the number of times the provider had previously encountered the alert. Shown are the total numbers of CKMB searches, CKMB searches associated with orders (both on the left vertical axis), and the proportion of searches not associated with orders (right vertical axis), grouped by the number of times that provider had previously searched for CKMB following alert implementation (and thus likely saw the alert). CKMB searches were rounded to the nearest integer

DISCUSSION

We demonstrate a strategy for tracking and analyzing CPOE user interaction data to evaluate the effectiveness of a test ordering alert and to provide insight into the mechanism of an alert's effect. In particular, we show that our CKMB alert was highly effective in reducing inappropriate test orders and that it accomplished this goal through a combination of just-in-time advice and longer term clinician education. Many prior studies have demonstrated the overall efficacy of various alerts and alerting strategies.[1-31418] However, to our knowledge, no prior studies have attempted to quantify the distinct advisory and educational effects of an alert. Thus, we considered the quantification of these effects to be an important aim for our study.

Our data have several implications for alert design and implementation. For example, the continued volume of CKMB-related searches as well as searches not resulting in orders weeks following the intervention highlights the need to communicate and advise clinicians on a policy changes over an extended time period. Reminders, either as CPOE alerts or other announcements, in the period immediately before and after a change, would likely be inadequate. In our study the need for an extended or even an indefinite alerting period may be in part a result of high provider turnover, particularly residents and fellows in multiinstitutional programs. While the issue of provider turnover may disproportionately affect hospitals where residents place many of the orders, even institutions with primarily full-time, permanent clinical staff may find value in a prolonged alerting period. In particular, our data demonstrating that even clinicians who had previously seen the alert still on occasion searched for CKMB without ordering it suggest the need for continued reinforcement. Nonetheless, one of the characteristics of an effective alert is that it should provide education and be triggered less frequently over time.

Several characteristics and features of our CPOE system and middleware were important contributors to the success of our alert and analysis strategy. Foremost, our analysis strategy would not have been possible without the CPOE capacity to track searches and orders along with associated information, such as providers, patients, and specific search terms used. Further, a well-designed and appropriately used CPOE system was essential. As previously described, efforts were made to develop an easy-to-use laboratory CPOE system that had robust search functionality.[17] The system design and implementation has created a culture and infrastructure such that free text orders are very rarely used. Since free text orders would not trigger the alert, frequent use of free text would have undermined our alerting strategy.

In addition, as previously described, a novel pathology middleware system interacts with our CPOE system, providing an interface that authorized pathology staff use to create and modify test ordering messages, alerts, and search parameters.[17] Using the middleware, development, and implementation of the alert was under the complete control of laboratory staff and took less than 10 minutes to create and test. Many alternative systems may utilize enterprise-wide information technology teams (central IT) to develop and implement CPOE alerts.[119] Central IT teams are outside the domain of pathology and typically have many competing priorities. Had our alert required development by central IT, the alert building process could have taken months (rather than minutes) and may have required approval by an “alerts committee,” despite CKMB guidelines having already been approved by the medical community. In addition, had our monitoring revealed that a change in the alert was indicated, or that expanded search parameters were needed to prevent free text, rapid revisions may not have been possible with a central IT approach.

Our analysis is subject to several limitations. As described in the methods, certain search terms were ambiguous, for example, a search for “cardiac markers” would most often be intended for multiple markers (i.e., CKMB and troponin), but occasionally, troponin alone might be desired. Our weighting system based upon preintervention search-order associations attempts to control for this uncertainty. Nonetheless, it is inexact and could overestimate both the postintervention CKMB search rate and the proportion of CKMB-associated searches not resulting in orders. For example, clinicians searching for “cardiac markers” may have more often intended troponin T without CKMB in the postintervention period than in the preintervention baseline period on which the weighting data are based. However, our data demonstrate a similar pattern when the analysis is confined to searches using terms unambiguously intended for CKMB (e.g., a search for “CKMB”). In addition, the data demonstrating that the rate of searches not associated with orders declines with each instance a provider sees the alert offer internal validation of our approach.

Another potential limitation stems from the use of pre- and postintervention comparisons without randomization. While randomization of the providers to either receive the alert or not receive it may have offered a better control group, doing so would have been impractical within the confines of our technology and institutional structure. Methods of policy communication besides the alert (e.g., email announcements and word of mouth) likely contributed significantly to the initial decrease in searches and orders. However, trends beyond this initial postintervention period were presumably primarily alert mediated. Furthermore, the proportion of searches not resulting in orders likely reflects mainly alert effects. We are not aware of any other changes between the pre- and postintervention period likely to confound our results.

In this report we apply this approach to only a single alert. However, the methodology described here is straightforward and should permit this analysis to be extended to a variety of alerts to better understand the characteristics important for success. In particular, our methodology of weighting search inputs permits a linkage to be established between searches and orders. This is an important aspect of the analysis since search is a central feature of many CPOE systems and without a systematic approach to determine user intent the connection between searches, alerts, and orders cannot be readily determined. The evaluation of a larger cohort of alerts may help more precisely define the alert characteristics most effective in offering advice and those most effective in educating.