Monte Carlo Simulations Demonstrate Algorithmic Interventions Over Time Reduce Hospitalisation in Patients With Schizophrenia and Bipolar Disorder.

Non-adherence with pharmacologic treatment is associated with increased rates of relapse and rehospitalisation among patients with schizophrenia and bipolar disorder. To improve treatment response, remission, and recovery, research efforts are still needed to elucidate how to effectively map patient's response to medication treatment including both therapeutic and adverse effects, compliance, and satisfaction in the prodromal phase of illness (ie, the time period in between direct clinical consultation and relapse). The Actionable Intime Insights (AI2) application draws information from Australian Medicare administrative claims records in real time when compliance with treatment does not meet best practice guidelines for managing chronic severe mental illness. Subsequently, the AI2 application alerts clinicians and patients when patients do not adhere to guidelines for treatment. The aim of this study was to evaluate the impact of the AI2 application on the risk of hospitalisation among simulated patients with schizophrenia and bipolar disorder. Monte Carlo simulation methodology was used to estimate the impact of the AI2 intervention on the probability of hospitalisation over a 2-year period. Results indicated that when the AI2 algorithmic intervention had an efficacy level of (>0.6), over 80% of actioned alerts were contributing to reduced hospitalisation risk among the simulated patients. Such findings indicate the potential utility of the AI2 application should replication studies validate its methodologic and ecological rigour in real-world settings.

Introduction

Approximately 600 000 people in Australia have severe mental illnesses such as schizophrenia and bipolar disorder.[1] Ongoing medication and psychosocial treatments are most often prescribed to patients with severe and serious mental illness[2,3]; however, the treatment and management of chronic mental illness is recognised as complex and is often silo-based and disconnected, particularly regarding the sharing of information between clinicians.[4] This shortcoming has led to major inefficiencies in the public health care system, poor management, and monitoring of patient’s quality of care in the community, and difficulty in ensuring adherence to treatment is maintained.

Indeed, adherence to treatment among severe mental illness populations is invariably low, with mean rates of only 42% in patients with schizophrenia and 41% in patients with bipolar disorder.[3] The most commonly reported non-compliance behaviours among these populations include refusing to comply with medical advice given or attend medical appointments, as well as incomplete or early discontinuation of a treatment programme.[5] Such defiance is associated with an adverse effect on the trajectory of the illness and leads to worsening symptomatology, higher rates of relapse, and recurring hospitalisation and ultimately increases in public health costs.[6] In fact, according to some estimates, over 80% of patients relapse several times within the first 5 years of initial treatment.[4] Non-adherence to treatment therefore presents as one of the most important challenges facing clinicians in treating highly prevalent psychiatric conditions. Indeed, it is clear that novel ways of enhancing treatment adherence are greatly needed as to reduce the prevalence of relapse and hospitalisation among schizophrenia and bipolar disorder populations.

The use of electronic health records (EHRs) has recently emerged as a potential approach to improve clinical decision making and patient outcomes.[7] In Australia, the MyHealthRecord system contains digital health records of all dispensing records of prescribed drugs, the Pharmaceutical Benefit Schedule (PBS), and records of all clinical services provided, such as medical appointments or laboratory tests conducted, the Medicare Benefit Schedule (MBS).

Evidence from a number of studies have shown that the use of EHRs has markedly improved our understanding of impending prognostic factors that are linked with recurring severe mental illness relapse and hospitalisation such as medication adherence, residual symptoms, concurrent physical health problems, and psychosocial difficulties.[8]

Computer algorithms in conjunction with using patients EHR data have been proposed as a means to accurately predict and quantify the risk of relapse and hospitalisation through a prediction model.[9] Algorithms systematically consider the combined effect of a set of prognostic factors, such as when people miss appointments, do not adhere to typical medication protocols, have unusual polypharmaceutical combinations, and have under or over testing.[9] Essentially, the algorithmic prediction model can estimate the level of risk an individual has in relation to future relapse. This type of tool can be used collaboratively by clinicians, health care providers, patients, and extended families when deciding on ongoing treatment and monitoring.

To date, there have been a handful of fundamental studies which have investigated the utility of computer-based algorithms in conjunction with EHRs on patient outcomes.[10] Vijayakrishnan et al[9] identified the signs and symptoms of heart failure (HF) among 50 000 primary care patients via exploration of their longitudinal EHR data. Specifically, retrospective analysis identified 4644 incident HF cases and 45 981 group-matched controls, demonstrating the potential for their novel data analytic tool to draw out rich data available within EHRs and apply it to predictive models for HF.

Lingren et al.[11] developed, evaluated, and validated a novel automated algorithm for detection of co-occurrence patterns of medical comorbidities in autism spectrum disorder (ASD) among a patient cohort from EHRs. Clustering analyses of comorbidities among the large cohort (N = 20 658 patients with ASD) accurately identified psychiatric, developmental, and seizure disorder clusters.

Similarly, Murphy et al[12] developed an EHR trigger algorithm to specifically identify delays in follow-up of patients with imaging results indicative of lung cancer. The trigger algorithm was applied to the records of 89 168 patients’ retrospective data, from which 131 were identified by the trigger as being high risk for delayed diagnostic evaluation. Further results confirmed a true absence of follow-up in 75 cases (trigger positive predictive value of 57.3% for detecting evaluation delays), with 4 subsequent diagnoses of primary lung cancer within the subsequent 2 years.

Using a hospital-based cancer registry from Osaka University Hospital, Gon et al.[13] conducted a study to validate an algorithm designed to determine stroke diagnostic code accuracy among 27 932 patients EHRs in Japan. Results revealed the diagnostic code and clinical examination combined improved the proportion of identified presence of disease in the diagnostic code, leading to improved accuracy and efficacy.

The findings above demonstrate the potential within the field of algorithmic-EHR interventions to improve patient care and well-being. In the light of this potential, this study aimed to evaluate the efficacy of an algorithmic intervention, namely, the Actionable Intime Insights (AI2) application,[14] in reducing reduced rates of hospitalisation among simulated patients with schizophrenia and bipolar disorder. The AI2 application sources Medicare Benefit Scheme (MBS) and Pharmaceutical Benefit Scheme (PBS) data from the electronic Australian MyHealthRecord registry from which it then stores in a temporal timeline trajectory. Using high-tech pattern recognition algorithms, the AI2 application is then able to identify and predict indicators of deterioration in functioning and subsequent risk of relapse/hospitalisation. For example, if the AI2 algorithm detects a gap between historically subsequent PBS prescription refills for a patient, and this exceeds beyond 61 days (effectively 2 months plus a grace period), then an alert is generated, stored, and used to notify appropriate care providers for potential follow-up with the patient.

The AI2 application has the potential of providing just-in-time adaptive early care intervention, which and can be shared simultaneously and jointly with both state-level and federal-level health care providers.[14] However, as the AI2 platform is yet to be tested in clinical practice, important questions remain to be answered. For example, the optimal frequency of alerts triggered over a period of time, the extent to which triggered alerts must be actioned, and in turn, how effective each actioned clinical intervention must be to reduce hospitalisation risks and improve patient outcomes is as yet unknown.

Therefore, the remaining of this article presents the development and demonstration of Monte Carlo simulation methodology to stochastically generate alerts for simulated patients with schizophrenia and bipolar disorder by the AI2 application that are generated over a 2-year period. The simulations are used to demonstrate the estimated, relative impact of actioning alerts at varying rates with clinical interventions of differing levels of effectiveness on hospitalisation rates. Five patient-simulated scenarios are described, each with different diagnoses and treatment protocol profiles.

Methods

Simulated patient profiles

Currently, there are several pharmacologic treatments typically used for patients with schizophrenia and bipolar disorder.[15,16] Typically, each treatment approach has a standardised protocol which stipulates how and when a patient should interact with their psychiatrist/psychologist and also pharmacist to receive the treatment. The characteristics of the 5 simulated profiles presented in this article represent 5 different hypothetical pharmacologic treatment interventions that could be used for patients with schizophrenia and bipolar disorder. The pharmacologic treatment types have not been chosen based on their demonstrated effectiveness, and their inclusion is not to suggest they are better or worse than other options. Merely, they have been chosen at random for the purpose of testing the utility and effectiveness of the AI2 algorithmic intervention.

The 5 simulated patient profiles below correspond to specific treatment plans that might be prescribed by clinicians to patients with differing levels of mental health issues, including frequency and type of medication and frequency of appointments. It is hypothesised that when a patient is adherent to their treatment protocol, the observed time period between consecutive appointments with the clinician, or the observed time period between consecutive visits to the pharmacist, matches the stipulated protocol:

Patient A (a patient with schizophrenia) is prescribed oral antipsychotics depot (long-acting injectable medication) by their treating clinician and has a follow-up appointment once in 6 months. They are instructed to pick up a prescription for medication from a pharmacy once a month.

Patient B (a patient with schizophrenia) is prescribed depot type D1 (long-acting injectable medication, for example, paliperidone) by their treating clinician and has a follow-up appointment once a month. They are instructed to pick up a prescription for medication from a pharmacy once a month.

Patient C (a patient with schizophrenia) is prescribed depot type D2 (long-acting injectable medication, for example, paliperidone Invega Trinza) by their treating clinician and has a follow-up appointment once in 3 months. They are instructed to pick up a prescription for medication from a pharmacy once a month.

Patient D (a patient with schizophrenia) is prescribed depot type D3 (long-acting injectable medication, for example, risperidone Consta) by their treating clinician and has a follow-up appointment once in 2 weeks. They are instructed to pick up a prescription for medication from a pharmacy once a month.

Patient E (a patient with bipolar disorder) is prescribed lithium treatment by their treating clinician and has a follow-up appointment once in 6 months. They are instructed to pick up a prescription for medication from a pharmacy once a month.

Using these simulated profiles as a baseline, the primary outcome of interest was the effect of the algorithmic intervention (parameter α) on the chances of readmission to hospital. In this model, α specifies the probability that a given alert will be actioned by a clinician, thereby leading to an intervention.

A secondary outcome of interest was the effect of the algorithmic intervention on clinical compliance, which was measured via the use of a secondary intervention (parameter β) that controls whether an intervention has a positive effect (β > 0), negative effect (β < 0), or no effect (β = 0). In addition, there was a tertiary parameter γ which specifies the probability that an intervention has any effect (either good or bad, depending on β); a non-effectual intervention corresponding to β = 0. For each value of γ the optimal value α* of the intervention/action rate was determined.

Simulation of alerts

Monte Carlo simulations were used to average 100 independent simulation runs. Specifically, for each run, 20 patients were selected from each of the 5 patient profiles above, to give 100 patients per run. Each patient profile specifies a treatment plan that determines the base rates of prescription refills and general practitioner (GP) appointment visits.

Five β values were chosen to represent the simulation complete in 1 working day. The specific values chosen were as follows: 2 for β > 1 (corresponding to a positive intervention effect), 2 for β < 1 (corresponding to a negative intervention effect), and β = 0 (corresponding to no effect from intervention). These values allow for verification that the simulation behaved correctly according to the mathematical models.

To simulate both compliance and non-compliance, the actual rates used for each patient in a given run were first randomised via a normal distribution centred on the base rate of each treatment plan with a standard deviation of 0.3. A randomised rate less than 0 or greater than twice the base rate was discarded and the personalised rate resampled.

For each patient, the simulator proceeded to stochastically generate the non-compliance events for 1 month at a time, sequentially over a 2-year span. Within each month to be simulated, the number of events of each type (eg, prescription refill, GP appointment visits, and hospitalisation readmission) was sampled from a Poisson distribution according to the corresponding effective rate. Hence, the base rates for refill compliance were chosen to correspond with the various treatment plans (ie, medication refill picked up once per month, corresponding to a Possion rate of ρ_0 = 1 refill/month).

The effective rate represents a per-month adjustment to the personalised rate based on the effect of events occurring in previous months, as described below. This per-patient simulation was repeated 10 times to produce 10 plausible event trajectories for each patient.

To avoid confounding the effectiveness of alerts with the detection of alerts, a basic assumption of this experiment was that non-compliance with the treatment plan specified in a patient’s profile could be automatically detected with 100% accuracy. In practice, the accuracy of detection remains to be tested in the real-time AI2 algorithm. Each such detection in any given month gave rise to an alert. However, the simulator used a stochastic mechanism such that each alert was only actioned by a clinical intervention with probability α. The value of this control parameter α was varied from 0 to 1 in increments of 0.05 for each simulation (of 100 runs).

It was further assumed that an intervention in 1 month (triggered by non-compliance) resulted in a temporary change in the rate of compliance in the next month. In practice, a variety of environmental circumstances mean that any given intervention could have either a positive or a negative effect on compliance. To model this, another simulation control parameter, β, was added to specify the multiplicative factor by which an intervention increased or decreased the probability of patient compliance with the treatment plan in the following month. Due to the amount of time it took to run the simulations, the values of β were limited to a discrete number of intervention factors, namely, moderately negative (0.8), weakly negative (0.9), neutral (1.0), weakly positive (1.1), and moderately positive (1.2). These values were chosen to be sufficiently distinct as to cover the desired range of behaviours. Other values tried in pretesting demonstrated similar characteristics.

In contrast to the effect of interventions, it was assumed that the rate of hospital readmission in any given month was temporarily increased for each consecutive month of non-compliance in preceding months. However, the counting of non-compliances was halted at any hospital readmission, on the assumption that the readmission involved treatment that would counter the effects of non-compliance in the same month as, or previous months to, the readmission.

The detection of non-compliance is heavily dependent on each patient’s treatment plan. For example, if the plan specified 6-monthly GP appointments, then non-compliance is indicated for a given month if there were no GP visits in that month or any of the preceding 5 months. Conversely, for fortnightly appointments, non-compliance was indicated if there were fewer than 2 GP visits in the month. For the purposes of this study, non-compliance was operationalised as no simulated PBS refills in any given month.

The simulator was implemented in the R programming language and run interactively in R Studio. In total, there were 100 runs × 5 profiles/run × 20 patients/profile × 10 repetitions/patient × 24 months/repetition × 21 α values × 5 β values = 252 000 000 distinct months of stochastic simulations. Computationally, the entire simulation took approximately 8 hours.

Mathematical models

All simulated treatment plans specify refilling the PBS medication script each month, corresponding to a base Poisson rate of ρR0 = 1 refill/month, giving the probability of 1 or more monthly refill event of

where ρR is the personalised randomisation of ρR0. An intervention modifies the probability of refill via

for a given intervention factor β;this latter probability, when inverted, corresponds to an effective refill rate of ρR′.

Each treatment plan also specifies a GP appointment every W weeks, where W depends on the particular plan. This corresponds to a base Poisson rate of ρV0 = 4/W visits/month, which is again randomised to obtain the personalised rate of ρV, giving

The intervention factor β similarly modifies the GP visit rate via

corresponding to an effective visit rate of ρV′.

The probability of hospital readmission depends on the patient profile. The appropriate baseline values are here taken to be 30% per year, for both patients with schizophrenia[17] on antipsychotic treatment and patients on lithium treatment.[18] This corresponds to a base Poisson rate of about ρH0 = 0.03 readmissions/month. The probability of one or more readmissions in a month is then

where ρH is again the personalised randomisation of the base rate ρH0.

The risk of readmission is worsened by the number of consecutive monthly non-compliances. It is estimated that a readmission probability of 30% per year increases to 50% after 6 months of non-compliance and to 80% after 12 months of non-compliance. A simple model approximately fitting these values is

for constant λ = 1.086, corresponding to an 8.6% increase in the probability of readmission for each month of non-compliance. Hence, the probability of one or more readmissions in a month is

The inversion of this probability gives the modified rate ρHn; for n = 0, it also gives the base rate ρH0. The effective, personalised, monthly rate of readmissions given n preceding, consecutive months of non-compliance is then taken to be ρH′ = ρH + ρHn − ρH0.

Results

Effect of intervention on compliance

Monte Carlo simulation runs were conducted to compute the average proportion of months in each patient’s simulated trajectory that contained events of interest. Results showed that for positive interventions (β > 1), the compliance rate increased with increasing α; for negative interventions (β < 1), the compliance rate decreased with increasing α; and for neutral interventions (β = 1), the compliance rate was independent of α. In addition, the average compliance rate of 0.39 was close to the theoretical value of 0.40 (see Figure 1). This moderate value is an artefact of the initialisation of the rates of the Poisson events models and could be increased by choosing initial rates that are biased upwards of the base treatment rates.

Figure 1.

The simulated average monthly compliance rate as a function of the probability α that a non-compliance alert leads to an intervention, for a variety of intervention quality effects: moderately negative, β = 0.8 (*); weakly negative, β = 0.9, (▽); neutral, β = 1.0 (o); weakly positive, β = 1.1 (△); and moderately positive, β = 1.2 (+).

Effect of compliance on readmissions

Figure 2 shows results of the plotted yearly probability of one or more hospital readmissions against the simulated compliance rate, for all values of α and β. As expected, the chance of readmission decreased as compliance increased, with observed readmission probabilities higher than the base value of 30%, which is likely due to the artefactually low rate of compliance noted above.

Figure 2.

The simulated average probability of one or more hospital readmissions per year as a function of the simulated monthly rate of compliance.

Effect of intervention quality

Figure 3 demonstrates the results of how effective the intervention quality was on actioned alerts. As expected, when all interventions were ineffective (γ = 0), then no alerts were actioned. Conversely, when all interventions were effective (γ = 1), then all alerts were actioned. The graph reveals that the success of interventions in reducing the risk of readmissions increases dramatically for γ ⩾ 0.6.

Figure 3.

The optimal probability α* that an alert should be actioned via an intervention (to minimise the hospital readmission risk) as a function of the proportion γ of interventions that are effective versus ineffective.

Discussion

Non-compliance with pharmacologic and psychosocial intervention treatments occurs frequently among people with schizophrenia and bipolar disorders, leading to subsequent increases in hospitalisation risk if undetected. In the light of this problem, the rationale of this study was to investigate whether an algorithmic intervention using the Medicare MBS and PBS data repository, specifically relating to prescriptions and appointments, could provide an effective alerting system that could detect when recommended treatment protocols are not being followed.

Stochastic simulation methodology was used to examine events of treatment non-compliance over time in patients with chronic severe mental illnesses and also observe the impact of the AI2 algorithmic intervention on reducing hospitalisation risk of actioning each alert of treatment non-compliance with an intervention. The investigated scenarios were chosen to represent the quality of clinical interventions for alerts varied in range from highly effective to ineffective to realistically represent variation expected in clinical practice.

Results indicated that the probability of patients being readmitted to hospital per year was higher among those who had lower monthly compliance with the intervention. This finding concurs with the results of Rosen et al[19] who found that patients with low or moderate medication adherence had significantly higher hospital readmission rates (20.0%) when compared with patients with high adherence (9.3%), and also the pooled findings of DiMatteo et al[20] meta-analysis, which found that of all medication-related hospitalisations that occur in the United States, between 33% and 69% are directly a result from medication non-adherence, translating to a subsequent cost of US $100 billion per year.[21] Collectively, these findings indicate that the criticality in developing innovative ways of assisting clinicians and hospitals identifies patients with non-compliance with treatment, who may be at risk for preventable relapse and hospital readmissions.

In relation to the effect of intervention quality, results demonstrated that when generated alerts from the AI2 algorithmic intervention were mild to moderately effective (between 0.2 and 0.6), around 10% of all actioned alerts effectively reduced hospitalisation risk. When the effectiveness of actioned interventions increased (>0.6), over 80% of actioned alerts from the AI2 intervention were contributing to effectively reducing hospitalisation risk. These results suggest that while poor-quality interventions following alerts had little to no effect on reducing hospitalisation, once the quality of actioned interventions reached an optimal level of effectiveness, the impact of the alerts on reducing hospitalisation rate increased rapidly.

From a clinical perspective, this suggests that even interventions of moderate quality following an alert could have a marked reduction in hospitalisation. Such findings demonstrate when real-time alerts are actioned into interventions, patients may benefit when they receive interventions of reasonable quality. An intervention of good quality seeks to recognise the patient’s biopsychosocial context, their situation perspective, and underlying motivations. It then skilfully shifts motivation and behaviour towards what is objectively in the patient’s best interest.[22] For example, adherence to medication along with any psychosocial supports that may be indicated. The patient’s self-management capabilities, competency of clinical practice, levels of care, and type of support and psychosocial options available are all potential factors contributing to the effectiveness of interventions in clinical practice.[23]

It is important to note the above findings in the light of several limitations. First, the simulated patients used in this study were based from hypothetical clinical Australian schizophrenic and bipolar disorder profiles. Hence, the question of whether the AI2-simulated model extrapolates to realistic outcomes cannot be answered at this point in time. For the ability to answer this question, and to improve on ecological validity and generalisability, future research in the area is pertinent, particularly replicable studies conducted with real-life patients and settings.

Second, the validity of the algorithmic intervention model is limited due to the fact that specificity and sensitivity analysis was not conducted. Thus, to be able to generate an accurate representation of how efficacious the alerts of the AI2 are in identifying patients at risk of relapse and hospitalisation, positive and negative predictive values will need to be established in follow-up studies. Finally, in this study, we assumed several factors affect the quality of clinical interventions patients receive without explicitly investigating effects associated with different factors. As mentioned above, one important factor to consider is the patient’s ability to self-care. It is unclear from these simulations whether patients or their clinicians should be alerted so that patients are best able to access effective and appropriate interventions. Real-world trials of this system in populations of patients with chronic severe mental illness could investigate whether effectiveness differs when alerts are sent to patients or their clinicians and carers or to both.

In summary, this study stochastically simulated events of treatment non-compliance in patients with schizophrenia and bipolar disorder over time and modelled how intervening in response to alerts of non-compliance could potentially reduce risk of hospitalisation. The findings from the analyses demonstrate monitoring and responding to alerts of patients’ changing levels of EHR treatment adherence detected in real time by algorithmic intervention methods that have potential in reducing hospitalisation risk.