Evaluation of computerized provider order entry systems: assessing the usability of systems for electronic prescription.

BACKGROUND: The field of medicine has been influenced by the growth and development of information systems such as the Computerized Provider Order Entry (CPOE) System.
OBJECTIVE: This study aimed to evaluate the usability of CPOE systems for electronic prescription in Tehran, Iran.
METHODS: This was an evaluation study conducted in 2017. The research population consisted of the CPOE systems used in hospitals of Tehran (Iran) and nurses who had access to, and used, the CPOE systems. Five hospitals with CPOE systems were included in the research sample. The data were collected using a questionnaire, and included a total of 50 questions. The questionnaires were distributed among 254 nurses who were the users of the systems. Data analysis was performed by IBM-SPSS version 21, using independent-samples t-test. A p-value of ≤0.05 was considered statistically significant.
RESULTS: Among the four aspects assessed, the "user-friendliness" (3.87±0.59) had the highest mean score. The lowest mean score (2.01±0.58) was related to the "decision support" feature of the systems. The highest and lowest mean scores for "prescription support" criterion belonged to system E (3.26±0.23) and system C (1.90±0.16), respectively. There was a statistically significant difference between the usability of the systems used in the private and the public hospitals (p<0.001). It was found that the CPOE systems in private hospitals had a higher level of usability (3.42+0.10) compared to those in public hospitals (2.91+0.25).
CONCLUSION: Two main functions of the studied CPOE systems i.e., decision support and prescription support should be developed to make electronic prescription safer and more intuitive. Addressing usability aspects of CPOE systems in practice could improve the usability of these systems for prescription.

1. Introduction

Nowadays, information systems are used in many daily activities and many of these systems have accordingly replaced the manual methods. The development of these systems has increased in recent decades, which makes the use of these systems inevitable (1–3). The field of medicine has also been influenced by the growth and development of information technology and one of the most important systems used in this area is the Computerized Provider Order Entry (CPOE) system (2–4). The CPOE system, as a key part of a clinical information system (CIS) enables the providers to issue orders electronically (5–7). Regarding the benefits of CPOE, this system has the capability of supporting physicians and other clinicians in the ordering process, and improving patient safety and the quality of care (4, 7, 9–11). A key aspect in developing information systems, including the CPOE systems, is focusing design efforts on users’ requirements and applying designing principles that ultimately can offer a more efficient and effective experience for users (12). Regarding CPOE, failure to address the designing principles can lead to consequences such as difficulty in the ordering process and risks to patient safety. There are indications that a CPOE system that lacks usability can increase medical errors and problems such as increased patient length of stay, increased costs and time-inefficiencies (13, 14). The usability evaluation of information systems is of particular importance to measure the quality of systems (15). The term “usability” refers to the capabilities of an information system in relation to what the system is expected to do and the ease with which the system can be used to support users in their practice (16). The evaluation of usability, therefore, focuses on measuring the usability requirements and identifying existing problems (15, 16). The usability evaluation of CPOE systems can provide valuable information for system developers, potential buyers, and the users (12, 17, 18). The evaluation of CPOE can address various aspects of this system, such as user-friendliness (the capability of displaying patient’s demographic and prescription information in a complete and readable mode, capability of using multiple screens simultaneously for various activities), control of drug orders (drug interactions, drug allergies, drug contraindications), generating alerts (the capability of detecting and tracking alerts, alerting drug contraindications and allergies), and capability of displaying all orders (including discontinued and interrupted ones) (16–19). Given the importance of this system for improving patient safety and care, this study was conducted to evaluate the usability of CPOE systems for electronic prescription in Iran, Tehran. In addition, in Iran the public hospitals mostly purchase their hospital information systems through the bureaucratic system of the medical university to which they are affiliated. However, private hospitals have a wider choice to select the system that they are buying. Therefore, a research hypothesis was set to examine whether the usability of the systems is different between the private and in the public hospitals.

2. Material and Methods

2.1. Study population and sampling

The research population of this 2017 study evaluation consisted of the CPOE systems used in general hospitals in Tehran and the nurses using the CPOE systems. The initial plan was to recruit clinicians including physicians and nurses for evaluating CPOE systems. However, when approaching the subjects in the hospitals, only in one hospital did both physicians and nurses use their CPOE system. In this hospital, a limited number of physicians agreed to take part in the study. In other hospitals, physicians did not use the systems due to either time limitations they faced or the systems lacked their expected functionality. As a result, only nurses were recruited for participation in the study. To maintain the confidentiality of the systems’ vendors and the hospitals, the identity of the vendors and hospitals were anonymized, and these are denoted as A, B, C, D, and E. To select samples of the systems, the vendors of the systems were initially identified in Tehran (5 vendors). It was then decided to choose a general hospital with the highest bed number from the hospitals that had prepared their system from one of the five vendors. Therefore, five systems were studied in five hospitals as follows: Systems A (420 beds), B (374 beds) and C (222 beds) in public hospitals and Systems D (200 beds) and E (120 beds) in the private hospitals.

2.2. Eligibility criteria

The inclusion criteria for involving nurses were: having work experience of more than 5 years, and having more than one year experience of using the CPOE systems. A total of 752 nurses were eligible in five hospital, and the Cochrane formula was used to determine the sample of study (n=254).

2.3. Study design and data collection

The data collection tool was a questionnaire that was developed mainly based on a validated CPOE usability questionnaire (20) and reviewing the literature (18, 19, 21–23), field observations and interviewing with nurses and the head of information technology departments of the hospitals. The questionnaire was composed of two sections. Section A included the demographic information of the participants, and section B was organized into four parts addressing usability aspects (patient safety, prescription support, decision support, and user-friendliness) with a total of 50 questions. The questions in this section were set based on the five-point Likert scale, ranging from the strongly disagree (score 1) to strongly agree (score 5). The content validity of the questionnaire was further checked by five experts in the field of medical informatics, namely two pharmacists, and three physicians. The reliability of the questionnaire was field-tested (Cronbach’s alpha = 0.92).

2.4. Data analysis

Data analysis was performed using IBM© SPSS© Statistics version 21 (IBM© Corp., Armonk, NY, USA). Independent-samples t-test was used to test the research hypothesis. The significance level was set at p≤0.05.

2.5. Ethical considerations

Participation of the subjects in the study was voluntary and performed in compliance with ethical consideration. Before data collection, a favorable ethical opinion was obtained from the Research Ethics Committee of Shahid Beheshti University of Medical Sciences (Ref: IR.SBMU.RETECH.REC.1395.1001). Also, the identity of vendors and hospitals were kept confidential.

3. Results

The response rate was 75% (n=193). Fifty-nine percent of the participants were female. With respect to academic degree, 76% of the nurses had a bachelor’s degree (n=147) and 21% had a master’s degree (n=45). In terms of participants’ age, the highest frequency was related to the 30–35 age group, while the lowest frequency accounted for 9.8%, which was seen in the age group of above 40 years. The average age of the participants was 32 years. Work experience of 60 % of nurses was less than 10 years (n=115) and the lowest frequency (11%) was for nurses with less than 20 years of work experience. In terms of experience of using the CPOE systems, 64% of the participants used the systems for 6–12 months (n=124), 22% of the nurses used CPOE for less than six months (n=43), and only 14% used the system for 1–3 years. Findings related to usability aspects are presented in the following sections.

3.1. Patient safety

The number of nurses in each setting was then calculated using Quota Sampling Method (Table 1). The highest mean score was assigned to system E (3.88±0.51) and the lowest score was related to system A (3.13±0.43). Systems D (3.68±0.20), B (3.50±0.46) and C (3.20±0.13) were in second, third, and fourth order, respectively (Table 1). With respect to system E, the highest mean scores were related to the capability of displaying patient medications at the time of prescribing (4.52±0.51) and the capability of making the final check of the prescription (4.47±0.59). Regarding system A, the lowest mean score was related to the capability of entering the cause of prescription (1.8±0.9) (Table 2).

Table 1

Number of eligible nurses and nurses involved in the study (calculated using Quota Sampling Method) and mean scores of usability aspects of CPOE systems

Hospitals (CPOE systems)	Number of eligible nurses	Number of involved nurses in each setting	CPOE Usability Aspects (Mean±SD)
			Patient Safety	Prescription Support	Decision Support	User Friendliness
A	280	95	3.13±0.43	2.40±0.53	2.03±0.61	3.42±0.39
B	145	49	3.50±0.46	2.59±0.20	1.83±0.32	4.24±0.37
C	105	42	3.20±0.13	1.90±0.16	2.07±0.97	3.72±0.62
D	125	33	3.68±0.20	3.23±0.43	2.24±0.36	4.33±0.74
E	97	35	3.88±0.51	3.26±0.23	1.98±0.26	4.09±0.18
Total	752	254	3.40±.047	2.59±.058	2.01±.058	3.87±.059

Table 2

Mean scores related to patient safety, prescription support, and decision support attributes of CPOE systems

Usability aspects		CPOE systems
		A	B	C	D	E
Patient safety attributes	Displaying patient medications when prescribing	4.07±0.65	4.4±0.49	4.43±0.47	4.4±0.5	4.52±0.51
	Asking for prescription signature before it is sent to the pharmacy	1.98±1.05	1.95±0.55	2.23±0.6	1.92±0.71	2.17±0.71
	Mandating prescriber to fill in the cause of prescription	1.8±0.9	1.7±0.44	1.75±0.44	2.5±0.75	1.91±0.79
	Enabling a prescription invalidation	4.1±0.55	4.25±0.44	4.31±0.47	4.3±0.54	4.39±0.44
	Mandating prescriber to sign for invalidation	1.85±0.85	1.5±0.55	1.8±0.61	1.7±0.65	1.86±0.69
	Enabling a prescription invalidation at the pharmacy	3.9±0.47	4.1±0.5	4.3±0.34	4.2±0.63	4.4±0.5
	Indicating that the invalidation is not made at the pharmacy	2.3±0.93	3.1±1.02	2.1±1.3	4.3±0.66	4.3±0.66
	Mandating prescriber to fill in the maximum daily dose	2.85±0.91	4.1±0.43	1.65±0.48	3.5±0.8	3.78±0.67
	Possibility of making the final check of a prescription	2.88±1	4.2±0.52	4.34±0.48	4±0.9	4.47±0.59
	Mandating prescriber to indicate the duration of treatment	2.68±0.94	1.61±0.57	1.65±0.48	3.6±0.78	3.6±0.78
	Displaying the name of a drug	4.2±0.45	4.44±0.5	4.53±0.5	4.17±0.61	4.47±0.51
	Displaying the dose of a drug	4.1±0.75	3.87±0.61	4.59±0.49	4±0.82	4.35±0.49
	Displaying the date of the prescription	4.14±0.5	4.3±0.46	4.56±0.50	4.39±0.49	4.35±0.48
	Displaying the name of a prescriber	3.87±0.94	3.93±0.56	4.09±0.89	4.17±0.66	4.21±0.67
	Making backup of prescriptions in the event of sudden shutdowns	2.76±0.81	4.6±0.6	1.81±0.69	2.32±0.54	4.1±0.63
	Indicating any disturbance in the electronic prescription service	2.67±1	3.74±0.56	3.84±0.76	4.14±0.59	3.35±1.02
Prescription support attributes	Renewing all medications with one key tap	2.09±1.07	3.79±0.51	1.78±0.61	4.07±0.71	3.48±0.99
	Availability of order set	3.07±0.90	3.83±.043	1.65±0.48	3.46±0.84	3.35±0.93
	Displaying generic drugs at the time prescription	3.38±0.72	4.19±0.53	4.28±0.46	4.36±0.48	3.69±0.82
	Displaying adverse drug event	1.79±0.74	1.70±0.63	1.84±0.67	1.92±0.85	2.43±1.03
	Displaying a duplicated drug	2.24±1.17	2.97±1.34	1.75±0.71	4.17±0.61	3.91±0.84
	Accessing to pharmaceutical database	1.88±0.83	1.51±0.51	1.62±0.49	1.71±0.71	1.87±0.69
	Displaying dosage error	1.79±0.69	1.49±0.58	1.69±0.64	1.75±0.7	2.26±0.96
	Displaying error in drugs administration	1.82±0.75	1.89±0.63	1.75±0.62	3.05±1.48	2.92±1.49
	Prescribing via a wireless network	2.01±0.85	1.57±0.5	1.66±0.65	3.75±0.88	3.43±0.94
	Displaying all orders, including current, stopped and interrupted	2.08±1.03	1.62±0.61	1.66±0.60	4.14±0.59	4.04±0.70
	Tracking errors and alerts	1.86±0.77	1.66±0.48	1.5±0.51	1.75±0.7	1.91±0.73
	Adding drug to discharge reports	4.01±0.58	4.06±0.75	1.72±0.58	4.21±0.63	4.26±0.45
Decision Support attributes	Generating alerts about interactions	2.83±1.34	1.72±0.45	1.59±0.49	1.82±0.67	2.48±0.79
	Generating alerts about allergies	1.62±0.75	1.45±0.5	1.63±0.49	1.68±0.72	1.83±0.65
	Generating alerts about contraindications	1.61±0.75	1.43±0.49	1.56±0.50	1.79±0.79	1.74±0.69
	Generating alerts about use of a drug during pregnancy	1.62±0.75	1.47±0.5	1.69±0.47	1.71±0.76	2±0.85
	Generating alerts about wrong prescription	1.62±0.75	1.47±0.5	1.63±0.49	1.61±0.63	1.56±0.5
	Generating alerts about laboratory test results	1.65±0.76	1.49±0.51	1.59±0.61	1.61±0.74	1.57±0.6
	Displaying the history of a drug prescription	1.67±0.8	1.49±0.51	1.05±0.51	1.57±0.57	1.83±0.72
	Integrating with other sub-systems of HIS	4.01±0.81	4.32±0.47	4.34±0.48	4.46±0.51	4.41±0.51

3.2. Prescription support

The highest mean score was seen for system E (3.26±0.23) and the lowest score was related to system C (1.90±0.16) (Table 1). Systems D (3.23±0.43), B (2.59±0.20) and A (2.40±0.53) were scored second to fourth, respectively. The highest mean score in system E was related to the capability of adding drugs to discharge reports (4.26±0.45) and the capability of displaying current and past medications (4.04±0.70). In system C, the lowest mean score (1.5±0.51) was related to the capability of tracking alerts generated by the system (Table 2).

3.3. Decision Support

With respect to the capability of systems in supporting decisions, system D was identified as having the highest mean score (2.24±0.36) and system B was indicated as having the lowest mean score (1.83±0.32) (Table 1). Systems C (2.07±0.97), A (2.03±0.61) and E (1.98±0.26) were respectively scored as second to fourth. Regarding system D, the highest mean score (4.46±0.51) was seen for the capability of interacting with other sub-systems of the hospital information system (HIS). Regarding system B, the lowest mean score (1.43±0.49) was related to the capability of warning about contraindications (Table 2).

3.4. User friendliness

Concerning user-friendliness of the systems, systems D (4.33±0.74) and B (4.24±0.37) obtained the highest mean scores, and system A (3.42±0.39) was indicated as having the lowest mean score (Table 1). Regarding system D, the highest mean score was related to the capability of the system’s interface in displaying texts in an easy to read format (4.35±0.48). With respect to the user-friendliness criteria, the highest mean score (4.24±0.43) seen in system B, was related to the capability of the system in providing users with the option to return to the previous screens. Concerning system A, the timesaving feature of the system obtained the lowest mean score (1.88±0.70). As shown in Table 1, among the four usability aspects, the highest mean score (3.87±0.59) was related to “user friendliness” and the lowest mean score (2.01±0.58) was for “decision support” aspects of the CPOE systems. A comparison among the four usability aspects of the CPOE systems is demonstrated in Figure 1.

Figure 1

Comparison of usability aspects of CPOE systems

3.5. Usability in private and public hospitals

The research findings also indicated that, there was a statistically significant difference between the usability of the systems used in the private and the public hospitals (p<0.001). According to the findings, the usability mean score was higher in the private hospitals (3.42±0.10) compared to the usability mean score in the public hospitals (2.91±0.25).

4. Discussion

The evaluation of the CPOE system is one of the approaches that can help to improve these systems. In terms of patient safety, more than half of the users noted that the CPOE systems had the capability of improving patient safety. The highest mean scores were related to displaying patients’ medications and enabling prescribers to have a final check of their prescription. These findings are supported by the study of Hagstedt et al. on evaluating CPOE systems, in which it was reported that patients’ medication information was displayed during prescription on the CPOE systems. In addition, the capability of invalidating a prescription was available on all systems except for one (16). In a study by Balina et al., 84% of users stated that the CPOE system improved patient safety (14). Similarly, the findings of other studies regarding the CPOE system also indicated that the use of this system resulted in improving patient safety (2, 20, 21). Improving patient safety reported in these studies appeared to be associated with the availability of information about patients’ medications when prescribing. Displaying this information together with the possibility of final checking of prescriptions could help to minimize or eliminate the risks associated with prescription errors. Gouyon et al. and Nuckols et al. highlighted that that CPOE can improve patient safety through minimizing the prescription errors (22–23). The findings of a study by Armada et al. showed that prescription errors significantly decreased after using the CPOE in the intensive care unit (24).

According to the findings of the current study, the CPOE vendors had addressed the patient safety considerations in the design of their systems. The patient safety aspects of the CPOE systems can be further improved by tailoring the systems in a way that the systems ask users to complete the cause of prescription and to sign the prescription before it is sent to the pharmacy. This checking process could help to make sure that patients are receiving the correct medications, and this could improve patient safety.

In relation to the prescription support, as findings indicated, this aspect had not been appropriately applied in the design of the systems. The highest degree of agreement among the criteria assigned to this feature was related to the capability of adding medications to the discharge reports. The highest degree of disagreement was related to the capability of tracking alerts and errors generated by the systems. A key factor related to prescription support capability of CPOE systems is the order set of these systems, which makes the ordering process more efficient (19). The findings of the current study indicated that the order sets of the systems were at an acceptable level, but further development is required. This was in line with the study of Chan et al., in which it was suggested that the order set of CPOE systems should be improved (25). Similarly, Khajouei et al. in their study reported that the order set of the studied CPOE systems needed further update to make prescriptions more intuitive (19). Overall, the prescription support of CPOE systems can be improved through improving the alerting capabilities and tracking alerts, as well as developing the order sets.

In relation to the decision support aspect of the CPOE systems, the findings showed the weakness of the systems, as more than half of the nurses noted that their systems did not meet this usability aspect (having the lowest mean score among four usability features). In the current study, the lowest mean score regarding the decision support aspect was reported for alerting about contraindications, as more than half of the nurses pointed out that the current systems lacked this capability. This is in line with the study of Eslami et al. who reported issues and shortcomings with the alerting capability of the studied CPOE systems. These researchers suggested a need for developing alerting functions of the systems (26). Similarly, Ferrandez et al. highlighted the importance of drug alerts and a need for improving this aspect of the CPOE systems (27). In another study, Hagstedt et al. indicated that the evaluated systems did not generate allergy warnings as appropriate (16). The issues found in the current study and reported in the studies noted above could be due to the limitations of, and inappropriate design of, the decision support systems of the reported CPOE systems. However, in a more recent study, Balina et al. reported that users were strongly satisfied with contraindications alerts generated by the CPOE systems (14), and this seemed to be due to developing the decision support capabilities of the studied CPOE systems.

Among the criteria related to decision support aspects of the CPOE systems in the current study, the highest mean score was reported for the capability of interacting with other sub-systems of hospital information, such as pharmacy information system and laboratory information systems. This is supported by Pevnick et al. who reported the improved data exchange between the radiology information system and the CPOE system studied (28). According to findings of the current study and studies noted above, paying attention to the decision support aspect of the CPOE systems, when designing or improving the systems, can help to minimize medication errors and improve patient safety. Possibilities such as sharing patient medication history and connecting systems to the drug information bank are among the approaches that can help to improve the decision support aspect of the CPOE systems. As a result, more efficient and effective use of the CPOE systems could be achieved.

With respect to user-friendliness, the majority of nurses pointed out that their systems benefited from this aspect. Among the four usability aspects, user-friendliness obtained the highest mean score. One of the key attributes related to user-friendliness criterion used in this study was the timesaving feature. However, half of the nurses noted that the CPOE systems did not help them to save time. This is in line with the study by Eslami et al. who reported that the use of some CPOE systems led to increased prescription time (26). This is also supported by the study of Vermeulen et al., in which nurses were concerned about time limitations that they faced using the CPOE systems, noting that the conventional method of prescription was indeed more efficient (29). However, there are studies indicating positive effects of CPOE systems. For example, in a study by Maat et al., the findings indicated a positive impact of the CPOE system on saving time when prescribing (30). Similarly, Balina et al. and Hoonakker et al. also showed that the use of CPOE systems led to saving prescription time (14, 2). The inconsistency of findings across the studies could be due to addressing the usability aspects for developing some systems and underestimating these aspects for other systems, as systems with higher usability can help users to perform their tasks more efficiently.

In relation to the research hypothesis, as findings indicated, the systems in the private hospitals had a higher level of usability compared to those in the public hospitals. In Iran, the Ministry of Health and Medical Education performs the accreditation and quality assurance of information systems, such as CPOE in the public sector. Therefore, in the public sector, the vendors are not accountable for improving the quality of their systems based on the end users’ needs, and the IT administration of universities are overloaded with other administrative duties and usually do not respond to all the detailed needs of the end users in a timely manner. In such an environment, the usability of the systems in the public hospitals is expected to be low. In contrast, in the private system, the hospitals pay for the information systems from their own revenue, and fully control the performance and functionality of the system, and the system vendor is accountable towards the hospital management and the end users. This could be one of the most important reasons behind the higher usability of the CPOE systems in the private hospitals compared to those in the public hospitals. Overall, considering the usability aspects when designing or selecting these systems, in particular the electronic prescription feature of CPOEs, could help to make efficient and effective use of the systems. In relation to existing systems, usability evaluation could help to improve systems capabilities through identifying usability concerns.

5. Limitations of the Study

There are limitations with this study. The first limitation is that physicians did not agree to take part in the study due to either their time limitations or not using the CPOE systems because of the systems’ limited functionality. The participation of the physicians could help to present a clearer picture of the usability aspects. The second limitation deals with the methodological approach to evaluation, as in this study only a questionnaire was used to assess the usability aspects of the CPOE systems. More in-depth findings might have been obtained if other approaches of usability evaluation, such as qualitative methods, had been applied.

6. Conclusions

The findings of the current study indicated that the evaluated CPOE systems should be developed, in particular in relation to prescription and decision support features to improve the electronic prescription and ultimately patient safety. The findings can be used by the suppliers of these systems in future developments. Knowing that there is no single best approach for evaluating the usability of systems such as CPOE, the application of mixed evaluation methods in future studies could help to obtain a better understanding of usability problems and take necessary actions.