Validation of a score to identify inpatients at risk of a drug-related problem during a 4-year period.

OBJECTIVE: Drug-related problems (DRP) produce high morbidity and mortality. It is therefore essential to identify patients at higher risk of these events. This study aimed to validate a DRP risk score in a large number of inpatients.
MATERIAL AND METHODS: Validation of a previously designed score to identify inpatients at risk of experiencing at least one DRP in a tertiary university hospital from 2010 to 2013. DRP were detected by a pharmacy warning system integrated in the electronic medical record. The score included the following variables associated with a higher risk of DRP: prescription of a higher number of drugs, greater comorbidity, advanced age, specific ATC groups and certain major diagnostic categories.
RESULTS: The study included a total of 52,987 admissions; of these, at least one DRP occurred in 14.9%. After validation of the score (period range, 2010-2013: 0.746-0.764), the area under the curve (AUC) was 0.751 (95% CI: 0.745-0.756).
CONCLUSIONS: This value is higher than those reported in other studies describing validation of risk scores. The score showed good capacity to identify those patients at higher risk of DRP in a much larger sample of inpatients than previously described in the literature. This tool allows optimization of drug therapy monitoring in admitted patients.

Introduction

Drug-related problems (DRP) have been defined as “an event or circumstance involving drug therapy that actually or potentially interferes with desired health outcomes” (PCNE, 2017). This is a general term that can encompass distinct terms referring to drug safety such as drug-related events, adverse reactions, and medication errors.

The DRP rates reported in the literature vary widely: figures for DRP as a cause of admission range from 2% to 10.3% (Angamo et al., 2017, Kalisch et al., 2012, Runciman et al., 2003, Singh et al., 2011, Pedrós et al., 2014), while in admitted patients ranges from 27.8% (Urbina et al., 2014) to as high as 81% (Blix et al., 2004), with various intermediate values reported by other studies (33 DRP por 100 pacientes ingresados (Bedouch et al., 2009), 58% (Dequito et al., 2011), 64.7% (Roten et al., 2010), 5.7–6.1% (Krähenbühl-Melcher et al., 2007).

This variability is likely due to the distinct terms used, as well as differences in types of hospital, the study population and age, type of admission and the methods used to identify DRP in published studies.

The use of specific drug groups during admission, such as opioids, diuretics, anticoagulants, antimicrobials and/or drugs of the cardiovascular system in general have frequently been implicated in the development of DRP in admitted patients (Bates et al., 1999, Bedouch et al., 2009, Bedouch et al., 2015, Blix et al., 2004, Davies et al., 2009, Krähenbühl-Melcher et al., 2007, Viktil et al., 2004). This is the result of greater complexity in certain diseases (Franz, 2012, Masoudi and Krumholz, 2003, Wong et al., 2011).

DRP among inpatients have been associated with high morbidity and mortality (Baena et al., 2014, Kongkaew et al., 2008, Nickel et al., 2013, Patel and Zed, 2002, Singh et al., 2011, Zargarzadeh et al., 2007). Thus, the mortality risk has been reported to be 1.88 (95% CI, 1.54–2.22) (Bates et al., 1997) during admission and to increase hospital stay by 1.91–4.6 days, (Bates et al., 1997, Classen et al., 1997) increasing costs by between $2262 and 4685 (Bates et al., 1997, Classen et al., 1997).

Several strategies have been associated with an increase in medication safety, such as the implementation of electronic prescription and some integrated clinical decision support systems. These have been associated with a decrease in the risk of medication errors and adverse drug events (Ammenwerth et al., 2008, Prgomet et al., 2017, Radley et al., 2013, Reckmann et al., 2009, Westbrook et al., 2015), as well as cost reductions (Ahmed et al., 2016, Eslami et al., 2008, Kaushal et al., 2006, Westbrook et al., 2015).

The identification of patients at higher risk of DRP is essential to allow closer monitoring of their drug treatment and to reduce their risk of experiencing a DRP (Davies et al., 2009, Khan, 2013, Parameswaran Nair et al., 2016a, Parameswaran Nair et al., 2016b, Zopf et al., 2008).

The implementation of the Computerized Physician Order Entry (CPOE) in the Hospital del Mar (Barcelona, Spain) was progressive, beginning in 2007. The Pharmacy Service developed in 2009 a score to identify inpatients at risk of a DRP and identified age, polypharmacy, greater severity as measured by the Charlson score, certain Anatomical Therapeutic Chemical (ATC) groups (https://www.whocc.no/atc_ddd_index/, 2017) and some major diagnostic categories as factors increasing the likelihood of a DRP during admission (Urbina et al., 2014).

Given that this score was obtained in a limited cohort of patients, the aim of this study was to test it in a larger cohort over a longer period of time.

Material and methods

Study design

The present prospective cohort study was performed in patients admitted to a 400-bed university hospital in Barcelona (Spain) during a 4-year period.

This study was approved by an independent ethics committee (Comitè Ètic d'Investigació Clínica del Parc de Salut Mar) (2016/6576/I).

No additional informed consent was required.

Study period

In a previous study, a score was designed, based on patients admitted to Hospital del Mar (Barcelona, Spain) in 2009 (Urbina et al., 2014). In the present study, the score was validated in a broader cohort of patients admitted during a 4-year period (2010–2013).

Setting

Tertiary university hospital with 431 beds (413 conventional beds plus 18 beds for critically-ill patients). The catchment area of the hospital has around 300,000 inhabitants living in two urban districts (http://www.bcn.cat/estadistica/catala/index.htm, 2017). The services provided by the hospital encompass acute medical and surgical care.

Patient population

To validate the score, the same exclusion criteria were adopted as those used to design the score to identify patients at risk of a DRP (Urbina et al., 2014). Thus, we excluded patients admitted directly to the critical care unit and/or those aged 18 years or younger. Likewise, admissions to the emergency department without hospital admission, or admission to the emergency department observation unit or resuscitation unit were also excluded because these units lacked the CPOE system in 2009.

The CPOE can be accessed by health professionals through the computerized medical records system of the hospital. Within the CPOE, there is a pharmacy DRP warning system that can be used by the pharmacy service. Both the CPOE and the DRP warning system have been previously described in detail (Urbina et al., 2014).

Drug-related problem-risk score

The present study used a previously designed score (Urbina et al., 2014). To design the score, data were used from patients admitted between January and August in 2009 to a tertiary university hospital (training set). The variables associated with having at least one DRP were identified by a multivariate binary logistic regression model and were used to compute the DRP risk score. This score was subsequently validated in a group of patients admitted between September and December 2009 (validation set). Currently, work is being carried out with the Informatics Service for its implementation in the CPOE and its use as a tool for the rapid and routine detection of DRP.

In agreement with the design of the score (Urbina et al., 2014), the following variables were significantly associated with the risk of DRP in inpatients: age older than 60 years (OR, 1.197), higher comorbidity (OR, 1.183), a higher number of prescribed drugs (OR, 3.335), diagnoses of some major diagnostic category (MDC) (Averill et al., 2007) and the prescription of drugs from certain ATC groups (Table 1).

Table 1

Variables associated with DRP.

Variable	OR (95% CI)	Points
Age > 60 years	1.197 (1.051–1.364)	1
Charlson index ≥ 2	1.332 (1.183–1.499)	1
Number of drugs during hospitalization > 10	3.335 (2.956–3.763)	3
MDC Others	1.393 (1.056–1.838)	1
MDC Nervous system	1.393 (1.002–1.937)	1
MDC Circulatory system	1.892 (1.400–2.557)	1
MDC Digestive system	1.393 (1.042–1.863)	1
MDC Musculoskeletal system and connective tissue	1.937 (1.432–2.619)	1
MDC Kidney and urinary tract	1.616 (1.169–2.235)	1
ATC C: cardiovascular system	1.546 (1.352–1.769)	1
ATC H: hormone therapy	1.198 (1.050–1.367)	1
ATC J: systemic, anti-infectious therapy	1.913 (1.696–2.157)	1
ATC S: sensory organs	2.559 (1.717–3.814)	2
ATC V: various	2.181 (1.679–2.834)	2

DRP, drug-related problem; MDC major diagnostic category; ATC, Anatomical Therapeutic Chemical classification system; OR, odds ratio.

Data collection

Admitted patients were classified according to whether they had a DRP or not during admission. DRP were identified by a team of clinical pharmacists through the CPOE.

Causes of DRP were considered according to the classification of the Pharmaceutical Care Network Europe (event or circumstance involving drug therapy that actually or potentially interferes with desired health outcomes).

A consensus was reached among clinical pharmacists on the identification of DRP raising the strongest doubts, in order to reduce bias.

The following variables were prospectively gathered for each inpatient: demographic data (age, gender), admission type (elective or urgent), MDC, admitting department (surgical or medical), comorbidity assessed with the Charlson index (Charlson et al., 1987), diagnosis-related group (DRG) weight (Averill et al., 2007), the number of different drugs administered during admission, and readmission (admission within 90 days of a previous admission). Drugs were classified according to the ATC classification system (https://www.whocc.no/atc_ddd_index/, 2017).

Statistical analysis

For the descriptive analysis of the sample, absolute and relative frequencies were calculated for categorical variables and the mean, standard deviation, median and interquartile range were calculated for quantitative variables.

A bivariate analysis of the data was conducted to confirm or refute the association between the presence of at least one DRP during admission with respect to each of the variables analyzed. For categorical variables, the chi-square test or the Fisher exact test was employed, when appropriate. For quantitative variables, the Student-test for independent variables or the non-parametric Mann-Whitney U test was used, depending on the normality of the distribution of the data. On the basis of the score designed in a previous study (Urbina et al., 2014), we calculated the risk scores for a DRP in each admission in a validation cohort. To determine the discriminatory capacity of the score, the area under the curve (AUC) was calculated, together with its 95% confidence interval, for the study as a whole and for separate years. The value of the marker defined as cut-off was determined by the maximum of the Youden index (J = sensitivity + specificity − 1).

A histogram is also presented of the percentage of DRP obtained in each risk range of the score.

Statistical significance was set at P < 0.05. The statistical analysis was carried out using the SPSS 18.0 statistical package (IBM Corp., New York, United States).

Results

There were 68,406 admissions between 2010 and 2013. After application of the exclusion criteria, the score was validated in 52,987 admissions corresponding to 34,672 patients (mean number of admissions per patient, 1.53 (Table 2). The mean age of admitted patients was 60.4 years (SD: 19.6) (range: 18–109) and 26,274 (49.6%) were men. At least one DRP was identified in 7882 (14.9%) of the admissions, the mean number of DRP being 1.44 (SD: 0.91) in these admissions (Table 2).

Table 2

Distribution of DRP, admissions, patients and prescriptions with DRP.

	2010	2011	2012	2013
DRP	3323	2812	3046	2174
Admissions	12,581	13,085	13,715	13,606
Admissions with DRP (%)	2245 (17.8)	1916 (14.6)	2132 (15.5)	1589 (11.7)
DRP per admission	1.48 (0.98)	1.47 (0.92)	1.43 (0.90)	1.37 (0.78)
Patients	9910	10,186	10,710	10,701
Patients with DRP (%)	1969 (19.9)	1674 (16.4)	1917 (17.9)	1442 (13.5)
DRP per patient	1.69 (1.30)	1.68 (1.20)	1.59 (1.17)	1.51 (1.00)
Prescriptions	133,214	138,551	149,410	148,721
Prescriptions with DRP (%)	3747 (2.81)	2918 (2.11)	3687 (2.47)	2662 (1.79)
Prescriptions per DRP	1.13 (0.36)	1.13 (0.36)	1.15 (0.39)	1.20 (0.43)

DRP, drug-related problem.

Of the 34,672 patients, at least one DRP was identified in 6327 (18.2%). In these patients, the mean number of DRP was 1.79 (SD: 1.49) (Table 2).

During the study period, 569,896 prescriptions were registered, with a mean of 10.76 prescriptions per admission (SD: 6.3). At least one DRP was found in 13,014 (2.28%) prescriptions (Table 2).

The total number of DRP was 11,355 (Table 2). The main reasons for these DRP were prescription errors due to incorrect use of the CPOE (19.3%), incorrect dose selection (18.9%) (drug dose too high (10.3%), drug dose too low (8.6%)), inappropriate drug-drug or drug-food combinations (11.5%), and dose adjustment according to renal function (8.2%) (Table 3).

Table 3

Distribution of the type of drug-related problem.

DPR	Total DRPn = 11,355 (%)
Prescription errors due to incorrect use of the CPOE	2188 (19.3)
Inappropriate combination of drugs, or drugs and food	1301 (11.5)
Drug dose too high	1174 (10.3)
Drug dose too low	976 (8.6)
Pharmacokinetic problem requiring dose adjustment (e.g. renal impairment)	931 (8.2)
Prescribed drug not available in the hospital formulary	869 (7.7)
Inadequate dosage regimen frequency	860 (7.6)
Inappropriate duplication of therapeutic group or active ingredient	546 (4.8)
Inappropriate drug form	306 (2.7)
Indication for drug-treatment not noticed	256 (2.3)
No evidence of effectiveness	210 (1.8)
Inappropriate drug (incl. contra-indicated or allergy)	179 (1.6)
No indication for drug	153 (1.3)
Excessive drug spectrum (overtreated condition)	136 (1.2)
Inadequate treatment duration	125 (1.1)
Inappropriate mode of administration	112 (1.0)
Inappropriate timing of administration	101 (0.9)
Inappropriate route of administration	53 (0.5)
Other DRP	879 (7.7)

CPOE: Computerized physician order entry.

Table 4 shows the bivariate analysis of the demographic and clinical characteristics of hospitalized patients with at least one DRP compared with those with no DRP.

Table 4

Bivariate analysis of the demographic and clinical characteristics of hospitalized patients with at least one DRP compared with those with no DRP.

	Admissions with DRPn = 7882	Admissions without DRPn = 45,105	p
Gender (%)
Male	4359 (55.3)	21,914 (48.6)	<0.001
Female	3523 (44.7)	23,191 (51.4)
Age (years)Mean (SD)	68.39 (15.49)	59.0 (19.9)	<0.001
Charlson index (%)
0	2351 (29.8)	23,167 (51.4)	<0.001
1	1784 (22.6)	8712 (19.3)
≥2	3747 (47.5)	13,226 (29.3)
Admission (%)
Urgent	5390 (68.4)	27,674 (61.4)	<0.001
Scheduled	2492 (31.6)	17,431 (38.6)
Admission (%)
Surgical	3249 (41.2)	26,476 (58.7)	<0.001
Medical	4633 (58.8)	18,629 (41.3)
Readmission due to prior admission (%)	1075 (13.6)	3394 (7.5)	<0.001
DRG weightMean (SD)	1.76 (1.60)	1.18 (0.96)	<0.001
Number of drugs during hospitalizationMean (SD)	16.24 (8.50)	9.80 (5.28)	<0.001
Major diagnostic category
Circulatory system	1410 (17.9)	5778 (12.8)	<0.001
Digestive system	923 (11.7)	6179 (13.7)	<0.001
Musculoskeletal system and connective tissue	877 (11.1)	3460 (7.7)	<0.001
Respiratory system	1251 (15.9)	5339 (11.8)	<0.001
Kidney and urinary tract	752 (9.5)	3451 (7.7)	<0.001
Nervous system	513 (6.5)	2911 (6.5)	0.855
Hepatobiliary system and pancreas	550 (7.0)	3284 (7.3)	0.338
Pregnancy, childbirth and puerperium	141 (1.8)	6206 (13.8)	<0.001
Other	1465 (18.6)	8492 (18.8)	0.598
ATC group
A: alimentary tract and metabolism	7555 (95.9)	40,830 (90.5)	<0.001
B: blood and hematopoietic organs	7322 (92.9)	39,648 (87.9)
C: cardiovascular system	6215 (78.9)	23,321 (51.7)
D: dermatological therapy	1518 (19.3)	3067 (6.8)
G: genitourinary therapy (including hormone therapy)	916 (11.6)	7450 (16.5)
H: hormone therapy	2983 (37.8)	9596 (21.3)
J: systemic, anti-infectious therapy	5661 (71.8)	21,148 (46.9)
L: antineoplastic therapy and immuno-modulatory agents	607 (7.7)	1727 (3.8)
M: musculoskeletal system	4354 (55.2)	26,830 (59.5)
N: nervous system	7176 (91.0)	40,449 (89.7)
P: antiparasitic products, insecticides and repellants	95 (1.2)	187 (0.4)
R: respiratory system	3512 (44.6)	9712 (21.5)
S: sensory organs	354 (4.5)	631 (1.4)
V: various	855 (10.8)	1.355 (3.0)

DRP, drug-related problem; MDC major diagnostic category; ATC, Anatomical Therapeutic Chemical classification system.

After validation of the score, it matched with the validation cohort (the AUC was 0.751 (95% CI: 0.745–0.756) with a cut-off of ≥5, sensitivity and specificity of 77.70% (95% CI: 76.77–78.62) and 60.98% (95% CI: 60.53–61.43), respectively, and Youden index of 0.39 (95% CI: 0.38–0.40).

Fig. 1 shows the rate of drug-related problems (DRP) in admissions according to the DRP-risk score in the study of the score design (Urbina et al., 2014), and the DRP rate in admissions according to the DRP-risk score in the present study.

Fig. 1

Rate of drug-related problems (DRP) in admissions according to the DRP-risk score in the study of the score design and in the present study.

Discussion

The aim of the present study was to validate a risk score previously designed in a representative sample. Thus, the score was validated in 52,987 inpatients. Validation of the score generated a ROC curve with an AUC of 0.751 (95% CI: 0.745–0.756) (range for the period 2010–2013: 0.746–0.764). This value is higher than those reported in other studies describing validation of risk scores (O'Connor et al., 2012, Onder et al., 2010, Parameswaran Nair et al., 2016b, Tangiisuran et al., 2014). Thus, three different studies reported AUC values of 0.73 (95% CI: 0.66–0.80) (Tangiisuran et al., 2014), 0.70 (95% CI: 0.63–0.78) (Onder et al., 2010), 0.67 (95% CI: 0.56–0.78) (Parameswaran Nair et al., 2016b) and 0.62 (95% CI: 0.57–0.68) (O’Connor et al., 2012).

A systematic review evaluated the quality of DRP prediction models in elderly inpatients (Stevenson et al., 2014). According to the results, the four studies included (McElnay et al., 1997, Onder et al., 2010, Tangiisuran et al., 2014, Trivalle et al., 2011) presented DRP prediction models of limited quality and their application in clinical practice was considered inadequate.

Given the limitations of the available models, a study is currently being carried out to develop a model to stratify adverse drug reactions in older patients admitted to hospital in the United Kingdom (Stevenson et al., 2016).

The score used in this study identified the following risk factors for experiencing at least one DRP: age older than 60 years, greater comorbidity, a higher number of drugs, specific MDC, and certain ATC groups as risk factors (Urbina et al., 2014).

Some of these factors have also been identified in other studies aiming to identify risk factors for medication-related events and even the development of predictive models for these events (Alhawassi et al., 2014, Onder et al., 2010, Stevenson et al., 2014, Tangiisuran et al., 2014, Zopf et al., 2008).

Although the rate of medication-related events is higher among the older population, their repercussion among inpatients in general is not inconsiderable (Bates et al., 1997, Classen et al., 1997). Therefore, the aim of our previous study was to design a risk score that could be applied in the majority of inpatients, similar to other studies that have evaluated risk factors for DRP (Bedouch et al., 2015, Blix et al., 2004, Classen et al., 1997, Zopf et al., 2008).

A constant finding in several studies is that a higher number of drugs is a risk factor for the development of DRP (Alhawassi et al., 2014, Angamo et al., 2017, Blix et al., 2004, Davies et al., 2009, O'Connor et al., 2012, Onder et al., 2010, Pedrós et al., 2014, Tangiisuran et al., 2014, Trivalle et al., 2011, Urbina et al., 2014, Zopf et al., 2008).

The DRP rate among inpatients in the present study was lower than that reported in other studies (Bedouch et al., 2009, Blix et al., 2004). Thus, a study of 8152 admitted patients identified a rate of 33 DRP per 100 admissions (1.71 DRP per patient) (Bedouch et al., 2009) while another study of patients hospitalized in the rheumatology and internal medicine units of 5 Norwegian hospitals reported that 81% had at least one DRP with a rate of 2.1 DRP per patient (Blix et al., 2004). A more recent systematic review, however, has indicated an extremely wide range of DRP prevalence rates among inpatients, oscillating between 0.03 and18 DRP per patient (Basger et al., 2014).

One of the most frequent DRP were prescription errors due to incorrect use of the CPOE. Its implementation has been associated with the appearance of a different and highly frequent type of DRP due to lack of familiarity with the complete set of applications of this computerized tool (Campbell et al., 2006, Koppel, 2005, Schiff et al., 2015). Thus, CPOE training courses are provided annually as a continual process for health professionals. In addition, a commission acted as a consultant for any questions users might have about the system. Other common types of DRP were interactions and overdosing, similar to the results of other studies conducted in French hospitals, in which the occurrence of interactions ranged between 12.6% and 16.7%, and overdosing between 19.2% and 12.8% (Bedouch et al., 2008, Bedouch et al., 2012).

One of the limitations of the present study is that validation was conducted in patients admitted to the same center. Because the characteristics of CPOE systems and their integrated applications may vary, this hampers extrapolation of the results to other centers and/or settings.

Another limitation is the variability in the study sample. In fact, the score could be more accurate in detecting DRP in patients with specific characteristics.

Moreover, patients admitted in the emergency department observation unit or resuscitation unit were not considered. This fact limits extrapolation of the results in this group of patients.

Conclusions

In the present study, the score was validated in a much larger number of patients than in previous reports and during a 4-year period, increasing the value of the score as a prediction tool.

Application of the score in hospitalized patients would help to target those that need closer clinical pharmacy monitoring because they are at higher risk of inpatient DRP. Given the constant patient turnover in acute-stay hospitals, the use of the score avoid missed DRP that could result in negative clinical consequences.

As in other studies, the score remains to be implemented in daily clinical practice. Its integration in the CPOE and is routine application in hospitalized patients will allow the identification of patients at higher risk of DRP.

Disclosures

All the authors have contributed to the study. This clinical study was carried out without any financial support and none of the authors have any conflict of interest to declare.

Conflict of interest

None.

Financial disclosure

Nothing to declare.