Predicting the risk of drug-drug interactions in psychiatric hospitals: a retrospective longitudinal pharmacovigilance study.

OBJECTIVES: The aim was to use routine data available at a patient's admission to the hospital to predict polypharmacy and drug-drug interactions (DDI) and to evaluate the prediction performance with regard to its usefulness to support the efficient management of benefits and risks of drug prescriptions.
DESIGN: Retrospective, longitudinal study.
SETTING: We used data from a large multicentred pharmacovigilance project carried out in eight psychiatric hospitals in Hesse, Germany. PARTICIPANTS: Inpatient episodes consecutively discharged between 1 October 2017 and 30 September 2018 (year 1) or 1 January 2019 and 31 December 2019 (year 2). OUTCOME MEASURES: The proportion of rightly classified hospital episodes.
METHODS: We used gradient boosting to predict respective outcomes. We tested the performance of our final models in unseen patients from another calendar year and separated the study sites used for training from the study sites used for performance testing.
RESULTS: A total of 53 909 episodes were included in the study. The models' performance, as measured by the area under the receiver operating characteristic, was 'excellent' (0.83) and 'acceptable' (0.72) compared with common benchmarks for the prediction of polypharmacy and DDI, respectively. Both models were substantially better than a naive prediction based solely on basic diagnostic grouping.
CONCLUSION: This study has shown that polypharmacy and DDI can be predicted from routine data at patient admission. These predictions could support an efficient management of benefits and risks of hospital prescriptions, for instance by including pharmaceutical supervision early after admission for patients at risk before pharmacological treatment is established. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Detailed and broad longitudinal prescription data from eight hospitals, allowing to delineate the daily prescription patterns for each hospital episode and identifying polypharmacy and drug–drug interactions.

Restricted set of features in prediction models which should already be routinely available or easily possible to implement in many hospitals.

Relatively low risk of information leakage and overfitting by testing the prediction performance in patients that were treated in another calendar year and by separating the study sites that were used for model training from the study sites that were used for testing the prediction performance.

No delineation of patient-specific benefit-–risk balances of prescriptions, for instance by including drug serum levels, results of electrocardiograms or individual pharmacogenetic risk factors.

Restriction to study sites from one large provider of inpatient psychiatric services in the region of Hesse, Germany.

Introduction

The most common medical decision is the prescription of medicines.1 Pharmacotherapy is also essential for the treatment of mental and behavioural disorders,2–6 where more than 130 different drugs with proven efficacy are currently available.7 The combination of multiple drugs is required in many clinical situations,8 but this is associated with an increased risk of drug–drug interactions (DDI),9 which enhances the risk of adverse drug reactions (ADR).10

The simultaneous use of five or more different pharmaceuticals is defined as polypharmacy.11 12 Its prevalence is high in both outpatient and inpatient settings, especially in old aged patients.13–17 Cost savings from reducing polypharmacy can be substantial.18 Due to an ageing population and an increasing number of patients with multimorbidity, polypharmacy is likely to become more frequent in the future, thereby further complicating the efficient management of benefits and risks of prescriptions in hospital psychiatry.19

DDI can be pharmacokinetic or pharmacodynamic in nature. In psychiatry, most pharmacokinetic DDI are related to the cytochrome P450 (CYP)-mediated drug metabolism.20 21 Pharmacodynamic DDI arise when the pharmacological effect of one drug is affected directly by another one.22 The latter is most relevant when combining drugs with anticholinergic or QT interval prolonging activity. Anticholinergic drugs decrease or block the actions of acetylcholine on central and peripheral acetylcholine receptors and may lead to multiple ADR like dry mouth, confusion, constipation, urinary retention, falls or delirium.23 24 Combining anticholinergic drugs increases the risk of the occurrence and severity of the respective ADR.24–26 QT interval prolonging properties of drugs can lead to ventricular arrhythmia (Torsade de Pointes, TdP) and cardiac arrest.27–30 Not all but most DDI are related to negative clinical outcomes and this relationship has been studied thoroughly. Therefore, for most drug combinations avoiding of DDI is a way to improve drug safety.

The use of information technology and the inclusion of hospital pharmacists are frequently discussed ways to improve efficient management of benefits and risks of prescriptions in hospitals. While research has shown ambiguous results for the effectiveness and acceptance of information technology to support continuous drug management during the hospital stay,31–35 systematic reviews have found improved patient outcomes for the inclusion of hospital pharmacists in adult36 and paediatric inpatients.37

There is a lack of evidence considering the potential to identify patients at risk of DDI at their admission to the hospital. If potential DDI risks can be detected before a patient’s pharmacological treatment is established, drug safety can improve by early focus on balancing patient-specific, combined benefit–risk ratios of all prescriptions. Furthermore, if polypharmacy can be predicted in advance, that is, detecting cases that will receive at least five drugs simultaneously, this allows early identification of patients that will have particular needs of drug safety management. The aim of the present study was to use routine data available at admission to the hospital in order to predict polypharmacy and DDI during the stay, respectively, and to evaluate the prediction performance with regard to its usefulness to support the efficient management of benefits and risks of drug prescriptions. A further aim was to compare the results of a machine learning approach with those achieved by means of logistic regression and by means of a naive baseline classifier.

Methods

Design, setting and participants

We carried out a retrospective, longitudinal study in eight psychiatric hospitals in Hesse, Germany. Our study included all inpatient episodes consecutively discharged between 1 October 2017 and 30 September 2018 (year 1) or 1 January 2019 and 31 December 2019 (year 2). Episodes at departments for child and adolescent psychiatry were excluded. An inpatient episode was defined as an individual patient’s stay at the hospital between admission and formal discharge with a planned duration of at least one complete day and night. Our study was part of a large clinical pharmacovigilance project sponsored by the Innovations Funds of the German Federal Joint Committee (‘Optimization of pharmacological treatment in hospitalized psychiatric patients (OSA-PSY)’, study number 01VSF16009).

Patient and public involvement

Patients and public were not directly involved in the design of this retrospective study.

Data

Our study used daily medication data from the electronic medical records at the study sites. These data contained each individual medication for each day of an inpatient episode. This allowed to investigate the medications given at each day separately and to include all modifications of pharmaceutical treatment during a stay. We defined polypharmacy as the concurrent prescription of at least five drugs per day.11 12 DDI were defined as (i) pharmacokinetic cytochrome P450 CYP-mediated DDI (CYP450-Interaction), (ii) the prescription of more than one concurrent anticholinergic drug (Antichol. Combi.) and (iii) the prescription of more than one concurrent QT interval prolonging drug (QT-Combi.).

Daily prescription data per episode were matched with DDI-information obtained from guidelines and additional studies. CYP-mediated drugs were identified in accordance to the Consensus Guidelines for Therapeutic Drug Monitoring in Neuropsychopharmacology38 restricted to clinically relevant drugs according to the Food and Drug Administration.39 In addition, melperone,40 levomepromazine41 and perazine42–44 were considered as CYP inhibitors. Additional non-psychotropic victim drugs were added based on CYP substrate properties defined by Hiemke and Eckermann.45 In total, these sources resulted in covering the following isoforms for analyses of CYP-mediated DDI: CYP1A2, CYP2B6, CYP2C19, CYP2C9, CYP2D6, CYP2E1, CYP3A4.

QT interval prolonging drugs were identified based on the lists of the Arizona Center for Education and Research, which maintains lists of drugs that have either a known or a possible risk for TdP.46 47 Anticholinergic activity of drugs was identified according to Hiemke and Eckermann, Chew et al and Lertxundi et al.45 48 49

The candidate feature variables that might potentially predict polypharmacy and DDI were obtained from routinely documented information in the electronic medical records and patient administration databases. A restricted set of feature variables was used that should be available in many hospitals at admission of patients. These were patients’ age at admission, gender, somatic and psychiatric diagnoses according to the 10th revision of the International Statistical Classification of Diseases and Related Health Problems-German Modification (ICD-10-GM), the treatment setting (inpatient vs day-clinic) and the scores at admission from the Global Assessment of Functioning Scale,50 the Clinical Global Impression,51 the Beck Depression Inventory52 as well as the Positive and Negative Syndrome Scale.53

Analysis

The study compared the results obtained through a machine learning approach with logistic regression and a naive baseline classifier. The chosen machine learning approach was gradient boosting with trees,54 as implemented in the CARET package in R.55 56 For comparison, we carried out a logistic regression with the same feature variables that were used in the machine learning approach. Furthermore, a naive baseline classifier was obtained by using only basic diagnostic groups in a logistic regression. The basic diagnostic groups were F0/G3 organic mental disorders/other degenerative diseases of the nervous system, F1 substance-related mental disorders, F2 schizophrenia, schizotypal and delusional disorders, F3 affective disorders and others.

We trained, validated and tested our models on different patients in order to avoid overly optimistic results from the evaluation of their prediction performance. Therefore, we divided data into a training set, that is, a random 80% sample of patients discharged in the first year, a validation set, that is, the remaining 20% of patients of the first year and a test set, that is, patients discharged in the second year. After separating the datasets, missing data were addressed by mean imputation and categories representing missingness of each individual feature.

We tuned hyperparameters on the basis of the trained models’ performance in the validation data set using the built-in grid search process in the CARET package, thereby modifying each of the four tuning parameters, that is, boosting iterations, max depth of trees, shrinkage and minimal terminal node size, until a maximum performance was reached in the validation sample.

The final models’ performance was assessed in the test dataset of episodes discharged in year 2. For judging variability of performance and robustness across different study-sites, we thereby trained eight different models of episodes discharged in year 1, each holding-out one study-site, and used these models to predict the outcomes of episodes discharged in year 2 from the held-out study site to restrict assessment of performance to hospitals not involved in the training process.

Prediction performance was compared using receiver operating characteristic curves (ROC) and Precision and Recall plots (PR-plots). We calculated 95% DeLong CIs for the area under the ROC.57 Furthermore, we defined different cut-off values for the operationalisation of the models that maximised sensitivity at a minimum precision of 0.6, 0.7 and 0.8. We chose a specificity threshold of 0.2 to be the minimum for a clinically meaningful application based on previous work of Tomašev et al.58 Furthermore, we defined a sensitivity of 0.2 as the minimum threshold for clinically meaningful application.

The results of machine learning models may be difficult to understand since the processes between input and output are opaque.59 This aspect is sometimes referred to the term ‘black-box’. In response, several methods have been developed to make machine learning models more interpretable.60 In our study, we calculated so called accumulated local effects of feature variables to show the average model prediction over the feature and to make their influence interpretable. Accumulated local effects describe the way features influence the prediction of a machine learning model on average.61 They are computed as accumulated differences over the conditional distribution and are considered unbiased even when features are correlated.62 The detailed methods for calculating accumulated local effects are described in detail in Molnar61 and Apley and Zhu.62

Results

A total of 53 909 episodes were included in the study. Table 1 shows the characteristics of the episodes in year 1 and year 2. The study periods used for training and validation (year 1) and for testing (year 2) were very similar in terms of both total number of inpatient episodes and their characteristics.

Table 1

Patient characteristics

		Year 1		Year 2
Number of episodes (n)		26 949		26 960
Study site (n and %)	1	1971	7	2039	8
	2	5977	22	6108	23
	3	2172	8	2157	8
	4	3432	13	3447	13
	5	5087	19	4874	18
	6	2710	10	2831	10
	7	3125	12	3073	11
	8	2475	9	2431	9
Day-clinic (n and %)		3444	13	3394	13
Female (n and %)		12 304	46	12 343	46
Age at admission (years, mean and SD)		47	18	47	18
Length of stay (days, median and IQR)		17	8–33	17	8–33
Diagnostic group (n and %)	F0/G3	2419	9	2330	9
	F1	8980	33	8791	33
	F2	4048	15	4086	15
	F3	8147	30	8259	31
	F4	1722	6	1650	6
	F6	1252	5	1347	5
	F7	185	1	265	1
	Others	196	1	228	1
	NA	0	0	4	0

IQR 25th–75th percentile.

Year 1: 1 October 2017 to 30 September 2018.

Year 2: 1 January 2019 to 31 December 2019.

F0, organic, including symptomatic, mental disorders; F1, mental and behavioural disorders due to psychoactive substance use; F2, schizophrenia, schizotypal and delusional disorders; F3, mood (affective) disorders; F4, neurotic, stress-related and somatoform disorders; F6, disorders of adult personality and behavior; F7, mental retardation; G3, other degenerative diseases of the nervous system.

Figure 1 shows the association between the maximum number of different prescriptions an episode has received per day and the proportion of episodes with a DDI. As expected, the likelihood of all DDI increased with the number of prescriptions. It appears remarkable that the risk of QT-Combi. showed the steepest increase between one and four different medications and levelled off afterwards. In contrast, the risk of CYP450-Interaction and Antichol-Combi. increased at a relatively constant rate. Further detailed results describing the frequencies of DDI and polypharmacy are provided in online supplemental figure S1 (for each diagnosis group) and the online supplemental figure S2 (for each combination of DDI and polypharmacy). The overall incidence of at least one DDI or receiving polypharmacy was relatively high (63%) and therefore an import aspect of clinical management.

Figure 1

95% CI of proportion of hospital episodes with drug–drug interactions versus maximum number of medications per day. CYP450-Interaction: pharmacokinetic cytochrome P450 (CYP)-mediated drug–drug interaction. QT-Combi.: a combination of at least two drugs on the same day with known or possible risk of Torsade de Pointes according to the Arizona Center for Education and Research classification. Antichol. Combi.: a combination of at least two drugs on the same day with at least moderate anticholinergic activity.

Figure 2 compares the possible combinations of sensitivity, that is, the proportion of correctly predicted actual positives, and specificity, that is, the proportion of correctly predicted actual negatives, that were reached by the different classifications. Furthermore, the operational points at the curves that maximise sensitivity at different minimum levels of precision are shown. Measured by the area under the ROC, the models for polypharmacy and DDI achieved a relatively good performance.

Figure 2

Receiver operating characteristic curves and precision and recall plots. Polypharmacy, receiving at least five different medications at the same day. A: precision at least 80%, B: precision at least 70%, C: precision at least 60%. Crossed circles show cut-off values that maximise sensitivity at different minimum thresholds of precision. Grey areas are not clinically meaningful because of a sensitivity or precision of less than 0.2. Dashed horizontal lines show the prevalence of the outcome. Diagonal lines show the random classifier bottom line. AUC, area under the curve; DDI, drug–drug interactions; GBM, gradient boosting machine.

The area under the ROC can be a potentially misleading measure of model performance when observations are distributed very unbalanced between classes.63 Therefore, figure 2 also compares the possible combinations of recall, a synonym for sensitivity, and precision, that is, the proportion of actual positives among all positive predictions. The analogous figures for the ROC and the PR-plots for each individual DDI are shown in online supplemental figure S3.

Figure 3 provides additional measures of classification performance based on a set of different potentially meaningful operational points. These should help to facilitate a more intuitively understanding of actual clinical usability. For instance, the model for polypharmacy operated at a minimum precision of 0.70 generated three false alerts for seven true alerts. It triggered alerts for one half of all admissions, thereby rightly identifying slightly more than three fourths of all actual positives and 72% of all actual negatives.

Figure 3

Measures of prediction performance. Polyphar.: receiving at least five different medications at the same day. Pr.: precision of at least. Prevalence: proportion of episodes with observed positive outcome. Trig. Rate: proportion of episodes that cause a positive prediction. True positive rate (a.k.a. sensitivity and recall): proportion of actual positives that are correctly identified as such. True negative rate (a.k.a. specificity): proportion of actual negatives that are correctly identified as such. False positive rate: proportion of actual negatives that are falsely predicted as positives. False negative rate,; proportion of actual positives that are falsely predicted as negatives. Pos. Pred. value (a.k.a. precision): proportion of actual positives in all positive predictions. Neg. Pred. value: proportion of actual negatives in all negative predictions. DDI, drug–drug interactions.

Two additional measures of model performance that are commonly used to evaluate the performance of clinical prediction models are calibration, that is, how consistent the predictions were with the observed rates, and the learning curve, that is, how the prediction performance increased with increasing number of episodes in the training data. These two measures are provided in online supplemental figures S4 and S5.

The overall prediction performance does not inform about the influence of individual feature variables on the predicted outcomes. These influences are not readily accessible in machine learning applications. Therefore, figures 4 and 5 show the accumulated local effects of the top eight features ranked by their variable importance in an effort to show the average model predictions across the features and make their influence interpretable. Variable importance is a dimensionless measure that represents the influence of each feature on the prediction performance relative to the other variables (the method is described in detail in Friedman54).

Accumulated local effects of top eight feature variables in predicting any drug–drug interaction. Comorb.: comorbidities. Age at adm.: age at admission. F10.2: dependence syndrome due to use of alcohol as main diagnosis. GAF at adm.: Global Assessment of Functioning at admission. PANSS NA at adm.: Positive and Negative Syndrome Scale missing at admission. Comorb. F10 (ICD): mental and behavioural disorders due to use of alcohol as secondary diagnosis. CGI at adm.: Clinical Global Impression at admission. BDI at adm.: Beck Depression Inventory at admission.

Accumulated local effects of top eight feature variables in predicting polypharmacy. Polypharmacy: receiving at least five different medications at the same day. Comorb. I* (ICD): diseases of the circulatory system as secondary diagnosis. Comorb.: comorbidities. Age at adm.: age at admission. Comorb. E* (ICD): endocrine, nutritional and metabolic diseases as secondary diagnosis. GAF at adm.: Global Assessment of Functioning at admission. CGI at admission: Clinical Global Impression at admission. Comorb. F10 (ICD): mental and behavioural disorders due to use of alcohol as secondary diagnosis.

The plots describe how the features influenced the prediction on average, with negative values representing decreasing probabilities and positive values representing increasing probabilities. Furthermore, the plots show the type of associations between outcome and feature variable, that is, whether they were linear, monotonic or more complex. The influence of feature variables on outcomes was as clinically expected, supporting the assumption that the models provided meaningful predictions.

Discussion

Key findings

The aim of the present study was to use routine data available at patient admission to the hospital to predict polypharmacy and DDI during the stay as well as to evaluate the prediction performance with regard to its usefulness to support the efficient management of benefits and risks of drug prescriptions. A further aim was to compare the results of a machine learning approach with those achieved by means of logistic regression and by means of a naive baseline classifier. The models’ performance, as measured by the area under the ROC, was ‘excellent’ and ‘acceptable’ compared with common benchmarks64 for the prediction of polypharmacy and DDI, respectively. The performance of the machine learning approach was very similar to the performance of the logistic regression models. Both models were substantially better than a naive prediction based solely on basic diagnostic grouping.

A reliable identification of patients at admission that will likely receive polypharmacy during their stay or are at risk of DDI would be beneficial for an efficient management of pharmacotherapy and the promotion of patient safety.65 These identifications would allow, for instance, the early involvement of pharmacists and specialised physicians that could supervise potentially complicated constellations and avoid DDI before a patient’s pharmacological treatment is established.

However, it is difficult to objectively define the prediction performance required for a useful application in routine clinical practice and this was out of the scope of the present study (see for instance66 67). Different applications and clinical settings likely require their own trade-off decisions between reducing false alerts and increasing coverage of actual positives, sometimes requiring intensive focus on the avoidance of missing actual positive cases. Our model was trained, tested and validated in the context of inpatient psychiatry. It has the potential to be useful in any clinical setting where polypharmacy and DDI are relevant and where the feature variables are readily available.

Regardless of prediction performance, machine learning techniques must be used responsibly in clinical practice, otherwise unintended effects can have severe consequences.68 The exact framework for a responsible use is currently under discussion.69–71 For instance, caregivers have to be trained in using the provided results, patients‘ access to care has to remain equitable, real-world performance must be constantly scrutinised, and responsibilities in case of errors have to be clear.

Furthermore, the effective implementation in clinical practice requires more than high prediction performance. Improved patient outcomes depend on how predictions are translated into effective decision making.72 This translation requires the predictions to be reasonably integrated in existing clinical processes in order to be accepted by medical staff and to create an actual benefit from better informed decisions.73

Present study in comparison to previous research

Previous studies found similar results considering the association between patient characteristics and the prediction of polypharmacy and DDI. Abolhassani et al studied 17 742 adult patients discharged between 2009 and 2015 from a department of internal medicine at s Swiss hospital.74 They found age, number of comorbidities and a higher Charlson Comorbidity Index independently associated with polypharmacy. Pérez et al studied 38 299 patients in 44 general practices in Ireland between 2012 and 2015 and found that age and multimorbidity were associated with a higher risk of DDI.75 Furthermore, they found that hospital admissions themselves were independently associated with a higher risk of DDI.

These studies did not investigate psychiatric hospital care. To the best of our knowledge, there is currently no evidence comparable in scale and scope investigating the prediction of polypharmacy and the risk of DDI in hospital psychiatry. Predicting other outcomes in hospitalised patients has often been found to be more complex in psychiatry than in other medical disciplines.76–78 Less distinct diagnostic concepts,79–81 less standardisation of care82 and a broader spectrum of acceptable therapeutic regimes83 were reasons put forward for this.

Several previous studies aimed to use patient and service data to predict outcomes of psychiatric hospital stays. These studies have often used a broad range of feature variables. Moreover, studies were often restricted to specific settings and patients. Leigthon et al84 attempted at predicting remission after 12 months in 79 patients with a first episode of psychosis applying a wide range of psychometric, demographic and socioeconomic feature variables and reached an area under the ROC of 0.65. Koutsouleris et al85 reached a sensitivity of 71%, a specificity of 72% and a precision of 93% in 108 unseen patients with their top 10 demographic, socioeconomic and psychometric features to predict remission in first episode of psychosis. Lin et al86 used single nucleotide polymorphisms from genetic analyses and clinical data and reached an area under the ROC of 0.82 in distinguishing responders from non-responders prior to antidepressant therapy in 455 patients with major depression.

Strengths and weaknesses of our study

A strength of this study is its detailed longitudinal prescription data. This allowed for delineating the daily prescription patterns for each episode and identifying polypharmacy and DDI. A further strength of this study is its broad coverage of psychiatric hospital care by including all inpatients admitted to eight psychiatric hospitals for 2 years. Furthermore, the study used a rather restricted set of features which should already be routinely available or easily possible to implement in many hospitals. Moreover, we were able to reduce the risk of information leakage and overfitting by testing the prediction performance of our models in patients that were treated in another calendar year and by separating the study sites that were used for model training from the study sites that were used for testing the prediction performance.

The present study did not include all potential types of pharmacokinetic and pharmacodynamic DDI that could occur under psychotropic medication. However, we have covered main types of DDI in hospital psychiatry, and our clinical prediction models will bring about a tool to support the efficient management of benefits and major risks of hospital prescriptions.

Our study did not delineate patient-specific benefit–risk balances of prescriptions, for instance by including drug serum levels, results of electrocardiograms or individual pharmacogenetic risk factors. Therefore, it was not possible to differentiate between DDI and actually inadequate prescriptions. Neither did our study document actual ADR. DDI do not necessarily lead to ADR and negative patient outcomes. However, the association between undesired DDI and an increased risk of negative patient outcomes has been thoroughly established by previous studies.10 24–30 Indeed, the association between DDI and actual negative patient outcomes might often be underestimated in psychiatry and therefore neglected in clinical practice.87

Another limitation of our study was the lack of time stamps on feature data. Patients were diagnosed and rated on clinical severity scales at admission and these values should mainly remained stable. However, we were not able to entirely rule out that these groupings might have been changed during the stay by clinical staff. A further general limitation of our study was its restriction to study sites from one large provider of inpatient psychiatric services in the region of Hesse, Germany. This obviously raises the question whether the prediction performance of the present models would remain stable if used in different healthcare systems or different clinical setting.

Conclusion

This study has shown that polypharmacy and DDI at a psychiatric hospital can be predicted from routine data at patient admission. These predictions could support an efficient management of benefits and risks of hospital prescriptions, for instance by including pharmaceutical supervision early after admission for patients at risk before pharmacological treatment is established. Future studies should investigate the clinical impact of such risk models on drug safety processes and patient outcomes in hospital psychiatry.