Development and Validation of 'Prediction of Adverse Drug Reactions in Older Inpatients (PADROI)' Risk Assessment Tool.

BACKGROUND: Adverse drug reactions (ADR) detection and prediction methods in hospitalized older adults remain imprecise. The identification of the risk factors for ADRs in this group of patients is crucial to develop plausible prediction models.
OBJECTIVE: This study aimed at developing and validating a "Prediction of ADR in Older Inpatients (PADROI)" risk assessment tool in hospitalized older adults. METHODS AND MATERIALS: We had previously conducted a derivational study that aimed to determine the risk factors of ADRs in hospitalized older adults. We developed the PADROI model as a potential ADR risk assessment tool incorporating 8 predictors each given a score by rounding off the respective adjusted odds ratios (AORs) to the nearest whole number. Subsequently, we conducted another prospective cohort among adults aged 60 years and older admitted to Gynecology and Obstetrics, Medical, Oncology, Surgery, and Psychiatry wards at Mbarara Regional Referral Hospital (MRRH) from July 5 to September 17, 2021.
RESULTS: A total of 124 participants, 70 females and 54 males aged 60-95 years, were included in this validation cohort; 62 of them experienced 90 ADRs. When applied to the derivational cohort, the area under receiver operating characteristic curve (AUROC) for the PADROI model was shown to be 0.896 (0.869-0.923; at 95% CI). In the validation study, AUROC of PADROI was 0.917 (0.864-0.971 at 95% CI; p < 0.001). Overall, PADROI correctly predicted 91.7% of those who experienced an ADR.
CONCLUSION: Using the adjusted odds ratios from our derivational cohort, we developed an ADR prediction tool (PADROI) that achieved an excellent AUROC (0.917), high sensitivity (87.1%) and specificity (90.3%). The current model demonstrated a high potential for clinical applicability which can be strengthened if similar results are reproduced in larger and multi-centered studies.

Introduction

The occurrence of adverse drug reactions (ADRs) continues to substantially contribute to the morbidity, mortality and high health care cost in the older population.1–4 The prevalence of ADRs among hospitalized older adults ranged from 6–46% in high income countries (HICs) and from 10.7–64.0% in low- and middle-income countries (LMICs). The majority (60%) of all ADRs in this patient population are potentially preventable.5

Current recommended practice for detecting and predicting ADRs in the elderly comprises thorough documentation and consistent evaluation of prescribed and over-the-counter medications through standardized medication reconciliation.6 ADR detection and prediction methods in hospitalized older adults remain imprecise. Focusing on high-risk medications and patients with multi-morbidity may advance prediction of adverse drug reaction.7

Obtaining a good ADR risk-prediction model usually occurs over four stages; first, identification of predictors of the phenomenon; second, validation that involves testing the potential model performance; third, evaluation of impact and usefulness in routine clinical practice; and lastly, implementation to assess acceptability and performance in real-life clinical practice.8

Area Under the Receiver Operating Characteristic Curves (AUROC) is the average value of the sensitivity for a test over all possible values of specificity or vice versa.9 In screening tests, sensitivity, the true positive rate, refers to the proportion of people that are correctly predicted to have a condition out of all people who actually have it. A sensitivity of 100% means the ability to correctly predict all people with the condition. Specificity, the true negative rate, refers to the proportion of those who were correctly predicted not to have a condition among those that actually do not experience it.10−13 Prediction models are categorized based on their AUROC curve values as: “excellent” (AUROC curve ≥0.900), “very good” (AUROC curve 0.80–0.89), “acceptable” (AUROC curve 0.70–0.79) and ”poor” (AUROC curve <0.7).14–16

Previously developed ADR risk-prediction models in older inpatients had achieved AUROC of 0.7017 to 0.7410 during their validation stages and all of them showed low specificity (<65%) and some of them used retrospective studies to develop their models.17,18 The existing models need further work to enable the development of a robust ADR risk-prediction model that is externally validated, with practical design and good performance.19 However, to the best of our knowledge, there is no study published on the validation of an ADR risk-prediction model for older inpatients in LMICs.

These ADR-risk prediction tools help health professionals to accurately predict patients that are going to incur an ADR during their hospital stay, thus, highlighting the need for close monitoring, avoidance of some medications or combination of drugs and to implement other relevant interventions and, ultimately, to mitigate the burden of ADRs in clinical as well as economic aspects.20,21 This study, thus, aimed at developing and validating ‘Prediction of ADR in Older Inpatients’ (PADROI), an ADR-risk prediction tool, by employing two separate prospective cohort studies in older adults in Uganda, a low income country.

Methods

Development and Scoring of the PADROI Model

The authors had previously conducted a systematic review on risk factors of ADRs among hospitalized older adults. It identified 15 independent risk factors reported by previous studies. The details of the systematic review were published elsewhere.5 Then we included the 15 previously reported risk factors and four more relevant independent variables, as independent variables in our derivational study conducted at MRRH from November 9, 2020 to May 7, 2021. This study, particularly, aimed at determining the risk factors of hospital-acquired ADRs in this group of patients. Details of the results were published elsewhere.22 In the current validation study, we aimed at developing a potential ADR-risk assessment tool from the data set of the derivational study, and then to validate it in a separate follow-on prospective observational study. The data on the 19 independent variables were extracted and were assessed for assumptions of logistic regressionand subjected to multivariable logistic regression. We first verified that there was no significant multicollinearity between any of the 13 independent variables. Moreover, both forward and backward conditional logistic regression similarly showed 8 out of the 13 independent variables to be significantly associated with the occurrence of hospital-acquired ADRs. Eight independent variables that were statistically significant and were retained in the final model included: age 60–75 (AOR = 1.97, 95% CI: 1.14–3.41; p = 0.015) compared with >75 years, previous ADR in 1 year (AOR = 2.43, 95% CI: 1.42–4.17; p = 0.001), PIM (AOR = 4.56, 95% CI: 2.70–7.70; p <0.001), polypharmacy (AOR = 3.29, 95% CI: 1.98–5.46; p <0.001), having CCI ≥6 (AOR = 8.47, 95% CI: 4.85–14.99; p <0.001), having heart failure (AOR = 2.83, 95% CI: 1.34–6.02; p = 0.007) or kidney disease (AOR = 1.95, 95% CI: 1.05–3.61; p = 0.034) and a hospital stay >10 days (AOR = 3.53, 95% CI: 1.89–6.61; p <0.001) compared with <5 days.

Validation Study

Study Setting and Period

This validation cohort was done at Mbarara Regional Referral Hospital (MRRH); the largest referral hospital in south-western Uganda. The hospital consists of Emergency, Inpatient and Outpatient services for a population of about four million people in its catchment areas including the districts of Mbarara, Bushenyi, Ntungamo, Kiruhura, Ibanda, Buhweju, Rubirizi, Mitooma and Isingiro. The hospital also receives patients from farther districts of Kabale, Masaka, Fort Portal and neighboring countries such as Rwanda and Tanzania. The current study was conducted in Gynecology and Obstetrics, Medical, Oncology, Surgery, and Psychiatry wards of the hospital from July 5 to September 17, 2021.

Study Design

A prospective cohort study was conducted to determine the prediction ability of the PADROI ADR prediction tool.

Study Population

We included all inpatients 60 years and older who were admitted to Gynecology and Obstetrics, Medical, Oncology, Surgery, and Psychiatry wards of MRRH during the study period and gave their informed consent. We excluded patients in coma or any level of unconsciousness as well as unstable psychiatric patients with mood disorders, schizophrenia and dementia who were on acute treatment. We also excluded patients who died or were discharged in less than 48 hours.

Sample Size Determination

The sample size for this validation study was calculated using MedCalc® Software Version 19.2 (©1993–2020), that employed the standard formula for sample size determination for ROC studies.23 Taking a power of 90% and Type I error of 0.01, null hypothesis AUROC (the best available AUROC in older inpatients) of 0.74,10 and a target AUROC of 0.896 obtained from derivational cohort of the current project,22 48 positive cases (patients with ADR) and 48 negative cases (patients without ADR) are required for this ROC study. By adding 25% for potential drop-outs, 60 positive cases and 60 negative cases, totaling 120 patients, who are 60 years and older were required.

Sampling Techniques

We employed a consecutive sampling technique involving all adults 60 years and older admitted at Gynecology and Obstetrics, Medical, Oncology, Surgery, and Psychiatry wards. Accordingly, all older patients who passed the inclusion and exclusion criteria were consecutively recruited until the target sample was achieved for both the cases and controls.

Data Collection

Three research assistants were involved in the data collection process. The principal investigator (senior clinical pharmacist) together with two physicians formed a team of experts. Firstly, an interviewer-based structured questionnaire was used to collect the data on participants’ characteristics, history of medical, surgical, gynecological and psychiatric conditions and previous drug uses, drug allergies, use of non-prescription and alternative medicines. Secondly, we reviewed patients’ medical files for diagnosis, comorbid conditions, previous adverse drug events, and diagnostic test results as soon as possible but within 48 hours. Thirdly, every day during their hospital stay except on Sundays, we updated the participants’ information after we interviewed them, conducted targeted physical assessments, and reviewed their medical files.

Fourthly, we used the British National Formulary (BNP)24 and UpToDate25 to identify the known ADR profile of the drugs used. We employed the Beers Criteria26 to identify PIMs. Polypharmacy was defined as concurrent use of five or more drugs (active pharmaceutical ingredients).18,27–31 Lexicomp® was used to detect clinically significant drug-drug interactions. The medications suspected for ADRs were classified according to the WHO-Anatomical Therapeutic Chemical (ATC) classification.32 Charlson Comorbidity Index was used to rate the complexity and prognosis of the participants’ conditions.33

All adverse events suspected by the principal investigator and the physician were considered for ADR causality rating and discussion by the team. The body systems affected by ADRs were categorized using the International Statistical Classification of Diseases for Mortality and Morbidity Statistics (ICD-11 MMS).34

Identification of ADRs

We used Edwards and Aronson’s definition of ADR: ‘an appreciably harmful or unpleasant reaction, resulting from an intervention related to the use of a medicinal product, which predicts hazard from future administration and warrants prevention or specific treatment, or alteration of the dosage regimen, or withdrawal of the product.’35 ADRs were first suspected when there was a relationship between the time of drug administration and the onset and course of the adverse reaction while excluding other potential causes.36 Every day except on Sundays, all ADRs suspected by the principal investigator were discussed with the team of experts consisting of the principal investigator (senior clinical pharmacist) and two senior physicians to establish the drug causality of the suspected ADRs. The principal investigator then used the Naranjo ADR assessment scale37 to rate the causal relationship of an ADR and the suspected medication. ADRs were classified as definite (9–12 points), probable (5–8 points), possible (1–4 points), or doubtful (0 points). We excluded all doubtful ADRs whereas we endorsed those rated as possible, probable or definite as’ possible ADR’s once consensus was reached by the team of experts. When consensus was not reached, a majority decision of the three members of the team was applied.

Data Analysis and Interpretation

The data were entered and cleaned by EpiInfo version 7.2.3.1 and then transferred to and analyzed by IBM Statistical Package for the Social Sciences (SPSS version 23.0 Inc., Chicago, Illinois). Descriptive statistics was used to determine the frequencies and percentages of the ADR occurrences and categories, drugs associated with the ADRs, and distribution of the predictors among patients with at least one hospital-acquired ADR and those without one.

For both the derivational and validation studies, we determined the predictive ability of the PADROI model by fitting Receiver Operating Characteristic curves and calculating the area under the curve and sensitivities and specificities at different cut-off points using ROC analysis using SPSS. A p value of <0.05 was considered statistically significant in all analyses.

Data Management and Quality Assurance

The research assistants were trained by the principal investigator. Then a pre-test was conducted involving two patients at each ward. Data collection procedure was revised based on the experiences from the pilot test. The collected data were checked daily for completeness and consistency by the principal investigator. Confirmation and causality rating of ADRs were discussed among the team of experts to reach a consensus. Data were double-entered, cross-checked, and password-protected.

Ethical Considerations

This study was conducted according to the Declaration of Helsinki.38 The current study was approved by Institutional Review Board of Mbarara University of Science and Technology (Reference No. MUREC 1/7) and Uganda National Council for Science and Technology (Reference No. HS992ES).

Results

Development and Scoring of the PADROI Risk Assessment Tool

A score was assigned to each of the 8 included predictors by rounding off the respective Adjusted Odds Ratios (AORs) to the nearest whole number. The score for hospital stay was considered only when patients were hospitalized for 11 or more days. For patients who experienced an ADR, the hospital stay referred to the number of days before the first ADR occurred instead of the total duration of hospital stay. Likewise, only polypharmacy and PIM incidents that were experienced before the occurrence of an ADR were used in scoring of PADROI. Charlson Comorbidity Index (CCI) calculated before the first incident of ADR was considered. Previous ADRs included any possible suspected ADR that occurred within one year preceding the current hospitalization. Kidney disease includes any structural or functional renal problem diagnosed by a doctor as confirmed by laboratory tests, diagnostic tools or biopsy. The weight of the risk associated with each independent predictor was obtained by rounding off the respective AORs to the nearest whole number. The sum of weights of the eight-risk factors of the PADROI model totaled 29. We also developed an alternative model by assigning points to each predictor using the adjusted β coefficients rounded off to one decimal and then by multiplying each score by 10 (Table 1).

Table 1

PADROI ADR Risk Prediction Score Form Derived from a Cohort of Ugandan Older Adults from July to September 2021

No.	Predictor	AOR of the Variable’s Category	Points Assigned According to AOR	β-adje	Points Assigned According to β-adj
1	Age 60–75 years	1.97	2.0	0.7	7
2	Previous ADR in 1 year	2.43	2.0	0.9	9
3	aPIM	4.56	5.0	1.5	15
4	bPolypharmacy	3.29	3.0	1.2	12
5	cCCI ≥6	8.47	8.0	2.1	21
6	Heart failure	2.83	3.0	1.0	10
7	dKidney disease	1.95	2.0	0.7	7
8	Hospital stay ≥11 days	3.53	4.0	1.3	13
Total		-	___/29	-	___/94

Notes: aPIM: Potentially inappropriate medication using 2020 Beer’s criteria; bPolypharmacy: The use of five or more different active ingredients of medicines; cCCI: Charlson Comorbidity index for 10 years survival; dKidney disease: any documented structural renal condition or eGFR<90 mL/min/1.73 m2; eAdjusted coefficients after being rounded off to one decimal.

ROC Curve for the Derivational Study

The AUROC curve for the PADROI model that was developed using adjusted odds ratios was shown to be 0.896 (0.869–0.923; at 95% CI; p <0.001) for the derivational study. This AUC is classified as’ very good or 0.800–0.899.’ The ROC curve was positioned at the top left for sensitivity which revealed its high prediction ability for ADRs (Figure 1). A cut-off point for PADROI score of 11 and above showed an optimal ADR risk prediction ability with a balanced sensitivity and specificity. At this point PADROI showed a sensitivity of 79.3% (true positive) and specificity of 86.1% (true negative). The mean PADROI scores were observed to be 15.5±5.8 and 5.9±4.6 for patients with ADR and those without respectively.

Figure 1

ROC curve of PADROI tool that was developed using adjusted odds ratios for the derivational study.

The AUROC curve was 0.897 (0.870–0.924 at 95% CI; p <0.001) for the PADROI tool that was developed using the adjusted β coefficients. Thus, the ADR risk prediction ability of the two tools was shown to be the same. Because of its simplicity (total scores of 29 compared with 94) and a smoother AUROC curve, we selected PADROI tool that was developed from the adjusted odds ratios for our validation study (Figures 2 and 3).

Figure 2

ROC curve of PADROI tool that was developed using adjusted β coefficients for the derivational study.

Figure 3

The comparison of ROC curves of PADROI tools that were developed using adjusted odds ratios and using β coefficients for the derivational study.

The Participant Characteristics

For this validation cohort, we studied 124 patients consisting of 70 females and 54 males; 62 patients with ADRs and 62 patients without ADRs (Table 1). The median age of the patients was 67 (62–75) ranging between 60 and 95 years. Fifty-one of the 124 patients were admitted at the Medical ward followed by 30 at Surgery wards (Figure 4).

Figure 4

The gender and ward distribution of older patients hospitalized at MRRH from July to September 2021.

ADR Occurrence and the Implicated Drugs

Sixty-two patients experienced a total of 90 ADRs during the current hospital stay. Applying the Naranjo ADR causality scale, 68 (75.6%), 19 (21.1%), and 3 (3.3%) ADRs were rated as probable, possible, and definite ADRs, respectively. ADRs affecting the nervous (38, 42.2%), gastrointestinal (28, 31.1%), and cardiovascular systems (13, 14.5%) were the three most frequently experienced ADRs during the current hospitalization. Metronidazole (11/90), ceftriaxone (7/90), furosemide (6/90), tramadol (6/90) and morphine (6/90) were observed to be the five drugs most commonly suspected as the cause of the ADRs (Table 2).

Table 2

The Types and Categories of ADRs Detected and the Implicated Drugs Among Older Patients Hospitalized at MRRH from July to September 2021

Category of ADR	Specific ADRs	Drugs Suspected for the ADRs
Nervous system: 38 (42.2%)	Dizziness (16), Drowsiness (15), Extrapyramidal reaction (2), Neuropathy (2), Headache (1), Lethargy (1), Metallic taste (1)	Metronidazole (8), Ceftriaxone (4), Furosemide (4), Morphine (3), Metoclopramide (2), Haloperidol (2), Tramadol (2), Ondansetron (2), Tramadol and Metronidazole (2), Haloperidol and Carbamazepine (1), Levofloxacin (1), Meperidine (1), Phenytoin (1), Spironolactone (1), Isoniazid (1), Bicalutamide (1), Digoxin (1), Clonazepam (1)
Gastrointestinal: 28 (31.1%)	Constipation (9), Nausea (5), Nausea and vomiting (4), Abdominal pain (3), Diarrhea (3), Hiccups (1), Xerostomia (1), Gastritis (1), Mucositis (1)	Ceftriaxone (3), Morphine (3), Capecitabine (2), Bisacodyl (2), Nifedipine (2), Docetaxel and Gemcitabine (2), Fluorouracil (2), Ondansetron (1), Amlodipine (1), Cisplatin (2), Codeine (1), Fentanyl (1), Furosemide (1), Metronidazole (1), Sevelamer (1), Tramadol (1), Vitamin C and Ferrous sulfate (1), Dexamethasone (1)
Cardiovascular: 13 (14.5%)	Tachycardia (4), Hypotension (3), Hypertension (3), Bradycardia (1), Fluid retention (1), Palpitation (1)	Carbamazepine (2), Omeprazole (2), Sildenafil (1), Furosemide (1), Dexamethasone (1), Ciprofloxacin (1), Carvedilol (1), Trihexyphenidyl and Haloperidol (1), Vitamin K (1), Haloperidol and Fluoxetine (1), Imatinib (1)
Endocrine & metabolic: 5 (5.6%)	Hypoglycemia (4), Hyponatremia (1)	Insulin (3), Metformin (1), Imatinib (1)
Dermatologic: 2 (2.2%)	Pruritus (2)	Dexamethasone (1), Vancomycin (1)
Eye/Otic: 2 (2.2%)	Visual changes (1), Tinnitus (1)	Tranexamic acid (1), Gentamycin and Tramadol (1)
Hematologic and oncologic: 1 (1.1%)	Anemia (1)	Cisplatin and Fluorouracil (1)
Renal: 1 (1.1%)	Renal insufficiency (1)	Cisplatin (1)

The Distribution of ADRs Among Patients with Different Risk Factors

Ninety-three (75.0%) of the patients were 60–75 years old; out of which 48 (51.6%) experienced at least one ADR. Among 98 (79.0%) patients who stayed in the hospital for 11 and more days, 52 (53.1%) incurred ADR. Similarly, among the 53 (42.7%) patients who had experienced an ADR in the previous one year, 33 (62.3%) experienced ADR during the current admission. Twenty-six (21.0%) of them had been diagnosed with a renal disease, 21 (16.9%) patients had heart failure and 37 (29.8%) had a CCI≥6. Another 45 (36.3%) patients took at least one PIM and 76 (61.3%) of the patients were on polypharmacy; out of which 37 (82.2%) and 49 (64.5%) experienced ADR respectively (Table 3).

Table 3

The Distribution of ADRs Among Hospitalized Elderly Patients with Different Independent Predictors and at MRRH, Uganda from July to September, 2021

Variables	Categories	Hospital Acquired ADR
		No	Yes	Total (N = 124)
		Frequency (%)	Frequency (%)	Frequency (%)
Age category	60–75	45 (48.4)	48 (51.6)	93 (75.0)
	>75	17 (54.8)	14 (45.2)	31 (25.0)
Hospital stay in days	≤10	16 (61.5)	10 (38.5)	26 (21.0)
	≥11	46 (46.9)	52 (53.1)	98 (79.0)
Previous ADR in 1 year	No	42 (59.2)	29 (40.8)	71 (57.3)
	Yes	20 (37.7)	33 (62.3)	53 (42.7)
Renal disease	No	54 (55.1)	44 (44.9)	98 (79.0)
	Yes	8 (30.8)	18 (69.2)	26 (21.0)
Heart failure	No	55 (53.4)	48 (46.6)	103 (83.1)
	Yes	7 (33.3)	14 (66.7)	21 (16.9)
CCI category	≤5	60 (69.0)	27 (31.0)	87 (70.2)
	≥6	2 (5.4)	35 (94.6)	37 (29.8)
PIM	No	54 (68.4)	25 (31.6)	79 (63.7)
	Yes	8 (17.8)	37 (82.2)	45 (36.3)
Polypharmacy	No	35 (72.9)	13 (27.1)	48 (38.7)
	Yes	27 (35.5)	49 (64.5)	76 (61.3)

Receiver Operating Characteristic (ROC) Curve of the Validation Cohort

The ROC curve for this validation study is positioned at top-left side showing a good prediction ability of the tool for hospital-acquired ADRs in hospitalized older adults. The current AUROC, which is the average value of the sensitivity for a test over all possible values of specificity or vice versa,9 of 0.917 (0.864–0.971 at 95% CI; p <0.001) is categorized as excellent (AUC >0.900). Overall, the PADROI tool has correctly predicted 91.7% of those who experienced an ADR but falsely classified 8.3% of the patients to be at no risk of incurring an ADR (Figure 5). The mean PADROI scores for participants with ADR and without ADR was shown to be 15.4±5.3 and 5.9±3.6, respectively.

Figure 5

The ROC curve of the validation study among older adults hospitalized at MRRH from July to September, 2021, Mbarara, Uganda.

Sensitivity and Specificity of the ROC at Different Cut-off Values for PADROI

As the scores of the PADROI increased from 0 to 10, the AUROC increased from 0.516 to 0.887 and its sensitivity declined from 100% to 87.1% and the specificity increased from 3.2% to 90.3%. Beyond the PADROI scores of 10, however, the AUROC did not show increase and then began to decline as the sensitivity was declining. Thus, a cut-off point for PADROI scores of ≥10 points showed an optimal prediction ability where it correctly predicted 54 (PADROI ≥10) out of the 62 patients who actually experienced an ADR (sensitivity of 87.1%) whereas 56 (PADROI <10) out of the 62 patients without ADR were correctly classified as not at risk of ADR (specificity = 90.3%) (Table 4).

Table 4

The Sensitivities and Specificities at Different Cut-off Points of PADROI Scores in Predicting ADR Among Older Adults Hospitalized at MRRH from July to September, 2021, Mbarara, Uganda

PADROI Scores	Hospital Acquired ADR			AUC	Sensitivity (%)	Specificity (%)
	No (N = 62)	Yes (N = 62)	Total (N = 124)
	Frequency (%)	Frequency (%)	Frequency (%)
Greater or equal to 1	60 (49.2)	62 (50.8)	122 (98.4%)	0.516	100	3.2
Greater or equal to 5	40 (40.0)	60 (60.0)	100 (80.6)	0.661	96.8	35.5
Greater or equal to 8	14 (19.7)	57 (80.3)	71 (57.3)	0.847	91.9	77.4
Greater or equal to 10	6 (10.0)	54 (90.0)	60 (48.4)	0.887	87.1	90.3
Greater or equal to 12	4 (7.1)	52 (92.9)	56 (45.2)	0.887	83.9	93.5
Greater or equal to 15	2 (5.1)	37 (94.9)	39 (31.5)	0.782	59.7	96.8
Greater or equal to 20	0 (0.0)	15 (100.0)	15 (12.1)	0.621	24.2	100
Greater or equal to 25	0 (0.0)	1 (100.0)	1 (0.8)	0.508	1.6	100
Total	62	62	124	*	*	*

Notes: *Not applicable. Bold: Recommended cut-off point.

Receiver Operating Characteristic (ROC) Curve of the Combined Cohort

The AUROC for the combined cohort was observed to be 0.900 (0.876–0.924 at 95% CI; p <0.001). It showed a sensitivity of 78.3% and a specificity of 89.4% at a cut-off PADROI point of 11 and above (Figure 6).

Figure 6

The ROC curve of the combined cohort of older adults hospitalized at MRRH from July to September, 2021, Mbarara, Uganda.

Discussion

Identification of the predictors of ADR occurrence in hospitalized older patients is crucial to develop a strong ADR-risk prediction tool.20,21 A systematic review of previously reported risk factors and a subsequent prospective derivational study were conducted to determine the risk factors of hospital-acquired ADRs in older adults. In the current study, we developed an ADR-risk prediction tool for older inpatients using adjusted odds ratios in the final model of multivariable logistic regression, and an alternative tool using the adjusted β coefficients of the same model. The two potential ADR-prediction tools showed the same prediction ability; AUROC curves of 0.896 and 0.897, respectively. However, because of its simplicity (total scores out of 29 compared with out of 94) and a smoother AUROC curve, we selected the PADROI risk assessment tool that was developed from the adjusted odds ratios for our validation study.

The PADROI’s AUROC curve of 0.896 of the derivational cohort is classified as’ very good or 0.800–0.899.’ The ROC curve was positioned at the top left for sensitivity which revealed its high prediction ability for ADRs. The current AUC is considerably higher than those reported during previous derivational studies including 0.623 in Irish older adults,18 0.74 in UK older inpatients,10 0.71 in Italian older patients,17 and 0.627 in Japanese older patients.39

While applying PADROI to the derivational cohort, a cut-off score of 11 and above out of 29 showed an optimal prediction ability for hospital-acquired ADRs in older adults. At this point, a sensitivity of 79.3% (true positive) and specificity of 86.1% (true negative) were observed. A cutoff point with a higher specificity compared with sensitivity (86.1% vs 79.3%) was preferred though falsely ruling out ADR risk has more deleterious effects on the patient. However, a sensitivity of about 80% is also quite adequate to minimize the number of cases falsely predicted to be at no risk. Higher specificity, in turn, enables to avoid unnecessary intensive monitoring for ADRs and needless avoidance of some medications though the ADR is not actually incurred. Unlike the previous similar ADR prediction studies that were shown to be unsuitable for a wide clinical use because of their AUROC values of less than 0.80,7 PADROI score showed a very good AUROC and was potentially applicable for wider clinical use. Thus, we carried out a separate validation cohort for external validation.

In the validation study, 62 patients experienced a total of 90 possible ADRs during the current hospital stay. ADRs affecting the nervous system (38/90), gastrointestinal (28/90) and cardiovascular (13/90) systems were the three most frequently experienced ADRs during the current hospitalization. The proportion of ADRs affecting the nervous system, however, is considerably higher than that of the derivational study. This can be explained by the inclusion of psychiatry wards in the validation cohort. On the other hand, these results are comparable with findings from previous studies that had shown ADRs affecting gastrointestinal tract,39–43 nervous system30,39,43 and cardiovascular system30,41 to be the commonest types in this population. Patients in psychiatry wards experienced more ADRs affecting the nervous system. Metronidazole (11/90), ceftriaxone (7/90), furosemide (6/90), tramadol (6/90) and morphine (6/90) were observed to be the five drugs most commonly suspected as the cause of the ADRs. These results are comparable with the derivational cohort and other previous studies that showed antimicrobials and drugs acting on the nervous system and cardiovascular drugs to be the most frequent culprit medications in hospital-acquired ADRs in older adults.40,43–47

Consistent with previous studies, the proportion of hospital-acquired ADR was higher in patients aged 60–75 years compared with those older than 75;41,42,48,49 among those who were hospitalized for 11 and more days, in patients with a history of an ADR in the previous one year,17,50 in those with a renal disease,18,22,30 in patients with heart failure17,18,51 and those with a CCI ≥6,17,18,48 in older patients that took at least one PIM18,30 and those on polypharmacy10,22,45,50 compared with their controls.

The ROC curve for the current validation study is also positioned at the top-left side showing high prediction ability of the tool for hospital-acquired ADRs in hospitalized older adults. AUROC measures the probability that’ a patient with an ADR had a higher predicted probability than a patient without an ADR.’ An AUROC of 1 represents a perfect model whilst 0.5 is random concordance.10−12

The current AUROC of 0.917 (0.864–0.971 at 95% CI; p <0.001) is categorized as excellent (AUROC >0.900). AUROC is the average value of the sensitivity for a test over all possible values of specificity or vice versa.9 Overall, the PADROI tool has correctly predicted 91.7% of those who experienced an ADR but falsely classified 8.3% of the patients to be at no risk of incurring an ADR.

Sensitivity is the ability of a screening tool to correctly identify people who have a condition whereas specificity is its ability to correctly identify people who do not have the condition.13 During the validation cohort PADROI scores of 10 and above showed the best-balanced sensitivity and specificity where it correctly predicted 54 (PADROI ≥10) out of the 62 patients who actually experienced an ADR to be at risk (sensitivity of 87.1%) and 56 (PADROI <10) out of the 62 patients without ADR were correctly classified to be at no risk of ADR (specificity = 90.3%). Thus only 12.9% and 9.7% of the patients were wrongly classified to be at no risk of an ADR and at a risk of ADR respectively. The current sensitivity and specificity of 87.1% and 90.3% are considerably higher than sensitivities and specificities of 80% and 55% in BADRI model from a UK study,10 68% and 65% in GerontoNet ADR Risk Score from an Italian cohort.17

The AUROC of the PADROI’s validation study is significantly higher than 0.592 from an Irish validation cohort,18 0.61 from a multi-centered observational study in Europe,52 0.623 in Ireland,30 0.70 in an Italian study,17 and 0.73 from another validation study of European patients.10 Our combined cohort, similarly, showed an excellent AUROC of 0.900 (0.876–0.924 at 95% CI; p <0.001) and a sensitivity of 78.3% and a specificity of 89.4% at a cut-off PADROI score of 11/29 and above. This is significantly higher compared with an AUROC of 0.566 reported by the combined cohort from an Irish study that applied the GerontoNet ADR risk scale to its combined cohorts.18

The possible explanations for the current higher AUROC include: higher number of variables studied in derivational cohort and thus more comprehensive final model as compared with the previous studies that consisted of 4–6 variables10,17,30 and the use of two separate prospective cohorts for derivational and validation studies as compared with ADRROP18 and GerontoNet17 models which were retrospectively developed from patient databases. Moreover, the lower AUROC curve from previous studies might be attributed to their inclusion of other study settings or different countries10,17,30 during the validation study unlike the PADROI model that was developed and validated in the same setting. Moreover, our study included all possible, probable and definite ADRs unlike O’Mahony et al. that excluded possible ADRs.30 This might explain the lower sensitivity (62%) reported by the latter.

The current validation study’s strengths include its power of 90%, consistency in ADR definition, ADR identification procedures, employing validated standard data collection tools, and engaging a multidisciplinary team consisting of doctors and pharmacists in an attempt to improve the accuracy of ADR identification and characterization. However, the study has some important limitations to be taken into consideration including: the model being driven from a single-centered study in a low income setting, external validation involving a single health facility that was the same as one involved in the derivational study and a short study period of the validation cohort. Accordingly, the types of medications used, the common medical conditions, and the health professional and health facility factors might differ from the conditions in other health facilities in the region and elsewhere. This might potentially limit its application in wider clinical uses and thus, we recommend larger and multi-centered cohorts of older adults to be undertaken applying this greatly promising model in low- and middle-income countries as well as high-income countries.

Conclusion

A prospective derivational study was conducted to develop an ADR-risk prediction tool named as PADROI among older inpatients. The PADROI model achieved a very good AUROC in derivational cohort, and excellent AUROC curve in validation cohort as well as in combined cohort. The PADROI model demonstrated a specificity of 90.3% and a sensitivity of 87.1%. The current AUROC curve, sensitivity and specificity were all considerably higher than those achieved by previous studies. We recommend a multi-centered validation study to evaluate the performance and impact the PADROI tool in clinical practice.