Predicting patient engagement in IAPT services: a statistical analysis of electronic health records.

BACKGROUND: Across England, 12% of all improving access to psychological therapy (IAPT) appointments are missed, and on average around 40% of first appointments are not attended, varying significantly around the country. In order to intervene effectively, it is important to target the patients who are most likely to miss their appointments.
OBJECTIVE: This research aims to develop and test a model to predict whether an IAPT patient will attend their first appointment.
METHODS: Data from 19 adult IAPT services were analysed in this research. A multiple logistic regression was used at an individual service level to identify which patient, appointment and referral characteristics are associated with attendance. These variables were then used in a generalised linear mixed effects model (GLMM). We allow random effects in the GLMM for variables where we observe high service to service heterogeneity in the estimated effects from service specific logistic regressions.
FINDINGS: We find that patients who self-refer are more likely to attend their appointments with an OR of 1.04. The older a patient is, the fewer the number of previous referrals and consenting to receiving a reminder short message service are also found to increase the likelihood of attendance with ORs of 1.02, 1.10, 1.04, respectively.
CONCLUSIONS: Our model is expected to help IAPT services identify which patients are not likely to attend their appointments by highlighting key characteristics that affect attendance. CLINICAL IMPLICATIONS: This analysis will help to identify methods IAPT services could use to increase their attendance rates. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Background

The 2017 Mental Health Foundation report1 concluded that two-thirds of people in England, Scotland and Wales say they have experienced a mental health problem. In 2008, the National Health Service (NHS) implemented the improving access to psychological therapies (IAPTs) programme across England. This programme now sees over 1.6 million new referrals per year2 with 193 services providing evidence-based treatments for people with depression and anxiety disorders, including children.3

The percentage of missed appointments in England is 12%, ranging between 2.5% and 25% across different IAPT services.4 Further to this, 42% of patients entering the IAPT programme only ever complete one treatment session.5 In order to complete a course of treatment, a patient has to receive at least two treatment sessions.6 These 42% of patients will thus never be considered to have completed a course of treatment, making it impossible to measure the improvement or deterioration in their mental health.

Like the rest of the NHS, those tasked with treating mental health are expected to do so under tight budgets and with limited resources. Many IAPT services have interventions in place to decrease the number of missed appointments, thus attempting to reduce the associated costs.7 In order to reduce the number of missed appointments, it is important to understand which patients are at higher or lower risk of not attending their appointments. These risks vary significantly between IAPT services and for patients at different stages of referral and treatment.2 4 Some patients will never engage with their therapy and thus not attend their first appointment, while other patients attend their first assessment and subsequently drop out of treatment at a later stage.

There have been previous studies into what influences a patient to miss their appointment. Hampton-Robb et al 8 investigated the influence of referral source and client income on whether a patient will attend their first appointment in the USA. Both factors were found to be good predictors of first session attendance and so we aim to use these in our study for England, noting that the two countries employ significantly different models of healthcare. In our research, we investigate the use of referral source in our model, but we are unable to measure patient income due to limitations of the IAPT datasets. We do have information on employment status and so use this as a proxy for income. Chan and Adams9 found there was limited variation in the rates of missed appointments for low and high intensity IAPT therapies. Their study was restricted by small sample sizes, whereas our study makes use of large amounts of historical data from services across England. Di Bona et al 10 also investigated predictors of attendance for IAPT therapy appointments, finding that risk, condition severity and self-referrals were predictive of attendance. Their study was similarly limited by small sample size. Jensen11 investigated the association between distance travelled to appointment and attendance. A further iteration of this work is to include similar investigations.

Non-attendance rates for first appointments were investigated by Murphy et al 12 who focused on the effect of beliefs and attitudes towards therapy. Their research found that on average 40% of first appointments are missed by patients, but this value varies from 16% to 67% across services in England. They found that attendance was not associated with patient age, gender or overall attitude towards therapy. Our study aims to use information from patients at the start of a referral, and does not have access to information on the patient’s attitudes towards therapy.

Objective

The objectives of this research are to determine the feasibility of fitting a generalised linear mixed effects model (GLMM) to predict whether a patient will engage with their therapy, and to predict whether a patient who has been referred to an IAPT service will attend their first therapy appointment. This type of modelling approach has only been used for patient outcomes and has not been used previously to model attendance in IAPT. Attendance rates for first appointments are much lower than for subsequent appointments,12 and so we initially focus on these.

Although similar research8–10 has been conducted regarding appointment attendance, our study utilises access to significantly more data. We are also able to investigate a wider variety of patient characteristics available when a patient enters therapy, differing from existing research.

Methods

Data

The IAPT programme ensures the same data are recorded throughout a referral regardless of the location of the given service, referred to as the IAPT dataset. This research includes data from 19 adult IAPT services. All patients who consented to their data being processed as part of the IAPT data set, were 18 years or over at time of referral and were referred between April 2012 and April 2019 from these services were included in the analysis. This totalled 959 100 referrals used in the analysis. Despite the consistent framework the IAPT dataset provides, each service, and even therapists within a particular service, can record certain information at different times during a referral.

The recorded dataset suffers from large amounts of missing data. To address this, multiple imputation (MI) was carried out at a service level, with five imputed datasets for each service. In order to impute a given variable, at least 50% of the data had to be present. We also assume the data are missing at random. The R package mice13 was used to perform the MI. Predictive mean matching was used to impute continuous variables such as patient age, polytomous logistic regression was used to impute unordered categorical variables such as employment status and logistic regression was used to impute binary variables such as the indicator for whether a service can send a patient a short message service (SMS).

We define our outcome as first appointment attendance. An appointment was not attended if it was cancelled by the patient, the patient arrived late and was not seen or they did not arrive. However, if the patient arrived on time, or late but was seen, the appointment was defined to be attended. If the appointment was cancelled by the provider, the appointment was flagged as happening in the future or the attendance code was not specified, then the data for that given appointment was not used in the modelling, this totalled 129 321 referrals.

Service level model

In order to determine which variables should be included in the GLMM, a multiple logistic regression was used to predict first session attendance for each of the services. The variables selected in these service level models were then considered as candidates for variables in the GLMM.

For each service, all variables used in the model selection process met a minimal threshold of association with appointment attendance (eg, a correlation coefficient of at least 0.1). Variable selection at a service level was performed using Least Absolute Shrinkage and Selection Operator in the glmnet R package.14 This type of model selection adds a penalty term to the residual sum of squares and shrinks the coefficients of the variables in the model towards 0 to see if this improves the fit. Variables are thus deselected from the model when the corresponding coefficient is 0.

The effects for each model were pooled over the five imputed datasets. The variables selected in each model, and the service to service variability in the estimated effects, were used when deciding which variables to include in the GLMM, and determining which should be considered random effects.

Generalised linear mixed effects model

The GLMM of attendance for patient i from service j is given by equation (1)

where the s are the coefficients of the fixed effects, the s are the random slopes for the random effects and .

To decide which variables should be random effects and which should be fixed, the variability of each effect in the service level models was first investigated. Those found to have a high variability in the magnitude of their effects were included as random effects, resulting in a random slope for each service j and factor level q. There is also a random effect of service, meaning there is a random coefficient for each service.

This model was fitted using the mgcv R package15 which is well suited to fitting GLMMs for large amounts of data using the bam function. The model was fit to the first 6 years of data for 18 services and tested with the most recent year from those services. This resulted in approximately 80% of data from these services being used to fit the model. This approach was chosen to investigate how frequently the model would need to be updated if used for attendance predictions within an IAPT service. The model was then validated with data from the remaining service.

Findings

Some variables were found to have more than 50% missingness for some services and thus were not imputed. Where imputed data were not available for all services, these variables were omitted from the GLMM, but included in service levels models where possible. The main variable of this kind was disability status (an indicator of whether a patient has a disability or not).

Preliminary analysis found that disability status and long-term condition (LTC) status were highly correlated when the data was recorded. As LTC status was generally better recorded than disability status, with fewer inconsistencies between services, we opted to use this in our GLMM.

Summaries of the variables used in the modelling can be seen in table 1, indicating the proportion of patients in each level of each categorical variable and summarising the numerical variables. For example in the training data, combined assessment and treatment is by far the most common appointment purpose (with an average of 88%), very few appointments are booked for the evening (average of 2.5%), around 60%–70% of patients seen are female, about half of patients are in employment and most (with an average of 90%) are not receiving statutory sick pay. We see that around 99% of patients have generalised anxiety disorder assessment as their appropriate anxiety disorder specific measure (ADSM). We also see that the proportion of missed appointments (average of 16.3%) is inline with, if slightly below the national average. We find that the test data are comparable to the training data.

Table 1

Train and test data sets. Minimum, maximum and average proportions for categorical variables. Average, SD, IQR continuous variables. For care professional role summaries see in online supplementary table 1

	Average (SD) for train set	Average (SD) for test set	IQR for train (test) set
Age at appointment	40.10 (15.10)	39.46 (15.56)	22 (23)
Time to appointment	14.03 (17.25)	14.18 (16.34)	15 (17)
Referral number	1.75 (1.27)	2.11 (1.82)	1 (2)

ADSM, anxiety disorder specific measure; LTC, long-term condition.

To identify which variables to include in the GLMM and specify random effects, we considered the magnitude of the effects of each selected variable in each service level model and the variability in these magnitudes across these models. We did not use data from the validation service for the model building. Variables whose estimated effects vary greatly by service were considered as candidates for random effects. In order to measure the variability of an estimated effect, we calculate a distance from the average per service, scaled by an average across all services. For categorical variables with levels 1, …, l, the distance from the average for each service is given by equation (2)

where is the estimated effect for level i, is the mean of the estimated effect for level i across all services. In order to compare distances between categorical and numerical variables, we used the OR per SD as a distance for numerical variables.

The variability in effects for each service is illustrated in figure 1 where each column of the heatmap represents one variable. Large variation of colour within a column (ie, very light and very dark present) indicates a large amount of variation in the distance from the mean effect for the given variable across the services. We see a lot of variation in referral source, appointment purpose, consultation medium, gender, LTC status, role of the care professional and patient’s ADSM. Due to the large amount of variation in these effects across services, we include random effects by service for these variables. All other variables in figure 1 show far less variation and are therefore considered fixed effects. For example, this means we allow the effect of referral source on attendance to vary between services whereas we assume the effect of time to appointment is the same.

Figure 1

Heatmap of the variability in the distances from the mean effects in service level models. Light colour indicates a small distance from the mean effect, darker colours indicate larger distances. ADSM, anxiety disorder specific measure; LTC, long-term condition.

The accuracy of the GLMM is determined from the test set. For this we must choose cut-off, p, for the predicted probability of attendance, where a patient is predicted to attend if their predicted probability of attendance exceeds p. ROC (Receiver Operating Characteristic) curve analysis was used to find the cut-off that minimises both the false positive and negative rates using Youden’s index. We found that a prediction cut-off of 0.82 resulted in a false attendance rate of 6.9% and a false non-attendance rate of 24.2%. Table 2 includes the confusion matrices and accuracy results for three cut-offs, 0.5, 0.75 and 0.82.

Table 2

Confusion matrices for prediction cut-offs of 0.5, 0.75 and 0.82 with accuracy, false attendance and non-attendance rates for each cut-off for the test dataset

Cut-off	0.5		0.75		0.82
	Reference
Prediction	Did not attend	Attended	Did not attend	Attended	Did not attend	Attended
Did not attend	3626	1715	12 283	17 675	18 930	40 523
Attended	26 827	135 095	18 170	119 135	11 523	96 287
Accuracy	82.9%		78.6%		68.9%
False attendance rate	16.0%		10.9%		6.9%
False non-attendance rate	1.0%		10.6%		24.4%

If we restrict our predictions to say only patients who have at least 82% chance of attending their appointment will attend, then the false attendance rate is minimised, the false non-attendance rate increases and the overall accuracy drops to 68.9%. An increase in the false non-attendance rate would result in more of the patients who would already attend their appointment receiving more encouragement to do so but would reduce the number of missed non-attenders.

Out of sample validation was performed with a holdout service. Using the optimal cut-off of p=0.82 found to minimise the false negative and positive rates in the test data, the model was 74.8% accurate with a false attendance and non-attendance rates of 21.1% and 4.1%, respectively.

Table 3 summarises the estimated effects from the GLMM. Focusing on the patient characteristics, we see older patients are more likely to attend their appointment with an OR of 1.02. Our model also shows that males are more likely to attend their appointments than females, although gender was generally not a significant predictor of attendance. Patients who are employed or are receiving statutory sick pay are more likely to attend according to our model. Patients suffering with agoraphobia may fear certain places or situations, our model shows that these patients are less likely to attend their appointments than those with other ADSMs.

Table 3

Estimated regression parameters, SE and p values for the GLMM. For care professional role summaries see online supplementary table 3

	OR	SE	P value
Intercept	4.18	0.57	0.01
Time to appointment	0.99	0.00	<0.001
Referral number	0.91	0.00
Age at appointment	1.02	0.00	<0.001
Time of day
Afternoon	Reference	–	–
Evening	1.04	0.03	0.11
Morning	0.99	0.01	0.08
SMS allowed
Not allowed	Reference	–	–
Allowed	1.04	0.01	<0.001
Employment status
Employed	Reference	–	–
Unemployed and seeking work	0.67	0.01	<0.001
Full or part-time student	0.98	0.01	0.20
Long-term sick or disabled	0.64	0.01	<0.001
Homemaker	0.67	0.01	<0.001
Not receiving sickness and disability benefits and not working	0.60	0.04	<0.001
Unpaid voluntary work	0.54	0.03	<0.001
Retired	0.73	0.02	<0.001
Not stated	0.64	0.04	<0.001
Statutory sick pay indicator
No	Reference	–	–
Yes	1.06	0.01	<0.001
Unknown	0.62	0.04	<0.001
Not stated	0.66	0.03	<0.001
Step intensity
Not stated	Reference	–	–
High intensity—first step	1.71	0.03	<0.001
High intensity—stepped up	0.75	0.03	<0.001
Low intensity—first step	1.39	0.03	<0.001
Low intensity—stepped down	0.07	0.03	<0.001
Referral source
GP	Reference	–	–
Other	0.89	0.12	0.33
Self	1.04	0.11	0.76
Appointment purpose
Assessment and treatment	Reference	–	–
Assessment only	0.86	0.37	0.68
Treatment only	0.98	0.31	0.94
Follow-up after left treatment	2.18	0.43	0.07
Review and treatment	0.86	0.33	0.65
Review only	1.77	0.37	0.12
Other	0.99	0.33	0.97
Consultation medium
Email	Reference	–	–
Face to face communication	0.59	0.42	0.21
SMS	0.48	0.48	0.13
Talk type for a person unable to speak	0.22	0.48	0.002
Telemedicine web camera	0.41	0.49	0.07
Telephone	0.54	0.42	0.14
Other	0.49	0.44	0.10
Gender
Female	Reference	–	–
Male	1.02	0.01	0.21
Not specified	1.59	0.22	0.03
Not known	0.69	0.37	0.33
Long-term condition status
No	Reference	–	–
Yes	1.01	0.12	0.95
Person asked but declined to provide a response	0.65	0.12	<0.001
Person asked and does not know or is not sure	0.65	0.13	<0.001
ADSM
Agoraphobia-mobility inventory	Reference	–	–
Panic Disorder Severity Scale	4.16	0.36	<0.001
Generalised anxiety disorder assessment	3.18	0.33	<0.001
Obsessive compulsive inventory	5.65	0.34	<0.001
Impact of events scale—revised	5.09	0.34	<0.001
Social phobia inventory	6.05	0.34	<0.001

ASDM, anxiety disorder specific measure.

In terms of service process characteristics, we find the longer a patient waits for their first appointment, or the greater the number of previous referrals they have had within the service, the less likely they are to attend. Furthermore, if the service is allowed to send the patient an SMS, the odds of attending increases by 1.01. Table 3 also shows that patients who self-refer to their IAPT service are more likely to attend their appointments than those who are referred by their general practitioners with an OR of 1.04. Other referral sources are seen to be worse attended with an OR less than 1.

We find that if a patient is in a transitional stage of being stepped up or down in intensity, they are less likely to attend this appointment with ORs of 0.75 and 0.07, respectively. We see however, from table 1, that there is some amount of variability in how IAPT services record step intensity.

When considering appointment characteristics, the consultation medium most likely to result in an attendance is email, however it is hard to define whether this type of appointment has been attended. We do see that a face to face appointment is more likely to result in attendance than the remaining consultation mediums. Further to this, appointments in the afternoon or evening are predicted to be better attended than morning appointments. As for appointment purpose, we see that review type appointments are more likely to be attended than others, but we note that appointments which are both an assessment and administer some kind of treatment are more likely to be attended than those that are only assessments or treatments.

Table 3 does not include results about the random effects. However, apart from the service random intercept, they were all significant predictors. We found that a few of the random effects are not normally distributed, but we do not need to be concerned about some departures from normality.16

Discussion

In this research, we have developed a GLMM to predict whether a patient will attend their first IAPT therapy appointment. The presented model includes random effects by service and service level random effects for appointment purpose, referral source, consultation medium, gender, LTC status, role of the care professional and patient’s ADSM. As well as fixed effects for these variables to capture the mean effect across the IAPT services, the model includes fixed effects for age at appointment, time from referral received to appointment, a flag indicating whether the service can send the patient an SMS, employment status, a flag for receiving statutory sick pay and the number of previous referrals a patient has had.

Of the appointment level characteristics, appointments scheduled for the afternoon are more likely to be attended. This association should be interpreted with caution because of the links between appointment time of day and employment status. Patients who are in employment make up the majority of the data and likely prefer appointments that fit around their work schedule, for example in the afternoon or evening. Modelling employed patients separately to those not in employment we find that an evening appointment is most likely to be attended when employed and least likely for the remaining patients. Future work is needed before the current model could be used to recommend appointment times most likely to encourage attendance. We also found that combined assessment and treatment appointments see reductions in the number of missed appointments. However, appointment purpose is unlikely to be communicated to patients beforehand. It should be also noted that some appointments may be recorded as assessment and treatment when only an assessment has taken place in order to meet waiting time targets.

A number of service process characteristics were highlighted to be associated with attendance. The rate of self-referrals and the number of patients consenting to receive an SMS can be influenced by the IAPT services by explaining the benefits to patients of receiving SMS appointment reminders and by encouraging self-referrals similar to Sims et al 17 and Di Bona et al.10 This study did not have access to data on whether or not a reminder was actually sent to the patient, which has been shown to improve attendance rates.18–20 An IAPT service may also be able to provide employment support before treatment to encourage patients to get back into employment. Appointments with a shorter waiting time from the date of the initial referral, and those for patients not in the process of being stepped up or down in intensity, were more likely to be attended. This suggests that IAPT services should aim towards more consistent care and shorter waiting times in order to improve attendance. Similar results have been demonstrated in Marshall et al.21

If we suppose that a patient will attend their appointment if their chance of attending is 82% or higher, then our model is 68.9% accurate with a false attendance rate of 6.9% and a false non-attendance rate of 24.4%. These results were calculated using the last year of data as the test data set, but the accuracy of the model was consistent if we instead used the last six or 3 months of data for testing. We would therefore recommend refitting the model every 6 months.

The out of sample validation resulted in high false attendance rates. We thus would recommend the use of this methodology for forecasting adult IAPT first session attendance using the services own historical data to refit the model rather than relying on the average effects.

Our model identifies a set of patient, appointment and service process characteristics most associated with attendance, which could guide services and clinicians towards patients who may require more encouragement to ensure their first appointment attendance. As an extension to this research we intend to perform further model validation with other IAPT services, explore the use of further measures of discrimination and to develop a model for predicting attendance throughout the duration of a referral.

Clinical implications

Getting patients into therapy is the first barrier for treatment. This analysis could help to identify which patients are likely to not engage with their therapy, meaning more effective intervention could be provided and also reduce overall costs. If the methods in this study were to be put into practice, it should be noted our model does not make use of potentially useful SMS and spatial data and relies on the use of historical data as it is intended for forecasting.