Measuring the impact introducing NHS 111 online had on the NHS 111 telephone service and the wider NHS urgent care system: an observational study.

OBJECTIVES: To explore what impact introducing the National Health Service (NHS) 111 online service had on the number of phone calls to the NHS 111 telephone service and the NHS urgent care system.
DESIGN: Observational study using a dose-response interrupted time series model and random-effects meta- analysis to estimate the average effect. SETTING AND PARTICIPANTS: NHS 111 telephone and online contacts for 18 NHS 111 area codes in England. NHS 111 telephone and online contacts data were collected between October 2010 to December 2019 and January 2018 to December 2019, respectively. PRIMARY AND SECONDARY OUTCOME MEASURES: Primary outcome: the number of triaged calls to the NHS 111 telephone service following the introduction of NHS 111 online. SECONDARY OUTCOMES: total calls to the NHS 111 telephone service, total number of emergency ambulance referrals or advice to contact 999, total number of advice to attend an emergency department or other urgent care treatment facility, and total number of advice to contact primary care.
RESULTS: For triaged calls, the overall incidence rate ratio (IRR) per 1000 online contacts was 1.013 (95% CI: 0.996 to 1.029, p=0.127). For total calls, the overall IRR per 1000 online contacts was 1.008 (95% CI: 0.992 to 1.025, p=0.313). For emergency ambulance referrals or advice to contact 999, the overall IRR per 1000 online contacts was 1.067 (95% CI: 1.035 to 1.100, p<0.001). For advice to attend an emergency department or other urgent care treatment facility, the overall IRR per 1000 online contacts is 1.050 (95% CI: 1.010 to 1.092, p=0.014). And finally, for those advised to contact primary care, the overall IRR per 1000 online contacts is 1.051 (95% CI: 1.027 to 1.076, p<0.001).
CONCLUSIONS: It was found that the NHS 111 online service has little impact on the number of triaged and total calls, suggesting that the workload for the NHS 111 telephone service has not increased or decreased as a result of introducing NHS 111 online. However, there was evidence to suggest an increase in the overall number of disposition recommendations (ambulance, emergency department and primary care) for NHS 111 telephone and online services combined following the introduction of the NHS 111 online service. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study is one of the first to look at how introducing NHS 111 online impacted the NHS 111 telephone service and the rest of the NHS urgent care services.

A dose–response interrupted time series model was used across 18 NHS 111 area code and used meta-analysis to estimate the average effect of introducing the online service.

As NHS 111 online was rolled out rapidly as a national service, it was not possible to have any control area codes. Therefore, any effects seen cannot be assumed to be the direct result of introducing the new service or whether they would have happened anyway due to other factors impacting both the NHS 111 telephone service and the wider emergency and urgent care system.

This evaluation was conducted at the early stages of implementation. Since COVID-19, public awareness and usage of the system may have changed.

The data and results for outcome relating to the wider NHS urgent care services only consider recommendations for care and not actual care received.

Introduction

National Health Service (NHS) 111 is a telephone advice and triage service for urgent but not life-threatening conditions.1 The NHS 111 service in England was set up in 2013 (previously NHS Direct) and aimed to improve patient access to urgent and emergency care services by directing the patient to the place most appropriate for their level of care need.2 This service is available 24 hours a day, 7 days a week. Initial triage is conducted by non-clinical call assessors. Where necessary, additional clinical assessment is provided by healthcare professionals. This service receives high volume of calls each year, but has had little impact on decreasing the demand for other NHS emergency and urgent care services.3 Data from NHS England show that approximately 15 000 000 telephone calls to NHS 111 were answered in 2017/2018; this equates to around 46 000 a day.4 This level of demand is also expected to continue to increase. For these calls, the proportions of final dispositions (recommendation on what to do next) are 21% for ambulance dispatches or Emergency Department (ED) attendances, 60.7% recommended primary care dispositions and the remainder to attend another service or self-care.5

In 2017, NHS England introduced four pilot NHS 111 online services as an alternative point of access for urgent care help. The online service allows users to use a web or ‘app’ platform to enter their health problem and then answer a set of questions to get advice on what to do next. At the end of their session a list of local services is provided, if needed a call back from a clinician at the NHS 111 telephone service can be requested or, where links allow, an appointment or call back from another service can be arranged.6 In 2018, this service was expanded rapidly to cover the rest of England using a single platform based on the NHS Pathways triage system used by the NHS 111 telephone service that is accessed by a webpage or the NHS app. The purpose of introducing the NHS 111 online system was to try to make the system easier and more accessible for users and to try to reduce the number of calls to the NHS 111 telephone service.1 A new service may reduce demand for another service by redirecting activity, but there is also a risk that demand is increased, either by duplication of service use or creating new demand.

Aim

This work is part of a larger mixed-methods study ‘Impact of NHS 111 online on the NHS 111 telephone service and urgent care system: a mixed-methods study’ which is published in the NIHR Health Services and Delivery Research journal.7 The larger study used both quantitative and qualitative methods to explore what impact introducing NHS 111 online had on the NHS 111 telephone service and the urgent care system. This article presents in greater detail the quantitative methods and analysis. Specifically, to investigate what impact introducing the NHS 111 online service had on the number of phone calls to the NHS 111 telephone service and the NHS urgent care system using a dose–response interrupted time series model.

Methods

Outcomes

Primary outcome

For this study, we looked at a number of outcomes. The primary outcome looked at what impact introducing the NHS 111 online service had on the number of triaged calls to the existing NHS 111 telephone service. Triaged calls are defined as any call that is assessed using the NHS Pathways triage/assessment process to determine what the health problem is, whether it is urgent and provide a ‘disposition’ advising what service is needed.8 This excludes any calls which, for example, just provide health information with no assessment.

Secondary outcomes

The secondary outcomes were used to explore the impact of NHS 111 online on the total number of calls and the various dispositions that could impact on other services.

Outcomes affecting NHS 111 telephone service only:

Total calls answered—including non-triaged calls for health information, those where the caller terminates before triage.

Outcomes affecting the wider NHS urgent care system:

Emergency ambulance referrals or advice to contact 999.

Advice to attend an ED or other urgent care treatment facility

Advice to contact or attend primary care.

The secondary outcomes, which looked at the impact NHS 111 online may have had on the wider NHS system, comprised of the combined dispositions from both the call and online data. For example, the outcome of emergency ambulance disposition is the combined number of 999 referrals from the telephone service and advice to contact 999 in the online data during the same time period.

Data collection

NHS 111 telephone and online contacts data were collected between October 2010 to December 2019 and January 2018 to December 2019, respectively. The NHS 111 Minimum Data Set, Time Series to December 2019 was used for the NHS 111 telephone data and was accessed from https://www.england.nhs.uk/statistics/statistical-work-areas/nhs-111-minimum-data-set/nhs-111-minimum-data-set-2019-20/ in January 2020.9 NHS Digital provided the NHS 111 online data. The telephone data were in the form of monthly counts at NHS 111 area code level, unlike the online data which were provided in the form of anonymous individual user sessions. Since the telephone data were at an aggregate level, all analyses were necessarily conducted at this ‘area code’ level.

Due to the level of the data, the NHS 111 telephone data were formed of records for 71 geographical ‘area codes’ and the online data were made up of Sustainability and Transformation Partnership (STPs) and Clinical Commissioning Group (CCGs) which could be mapped to 38 NHS 111 area codes. (The remaining telephone data NHS 111 area codes were old codes that have been subsequently merged into some of the newer area codes). These area codes are made up of STPs which are in turn made up of CCGs. Unfortunately, area codes and STPs are not coterminous; area codes can be formed of multiple STPs and some STPs can be split over more than one area codes. Given that the online 111 service was introduced at STP level, this meant that not all NHS 111 area codes could be included in the analyses.

In addition, for the interrupted time series analysis one full year of NHS 111 online data were required, therefore any area codes where the online service had not been operating for at least a year were removed. For consistency, the telephone data were capped to 2 years prior to the introduction of the NHS 111 online service. This meant each NHS 111 area code had a minimum of 36 months of data. In total there were 18 NHS 111 area codes remaining for the analysis. A list of the 18 sites and their CCGs that were included are provided in the online supplemental appendix table 1.

As the telephone data provided for this study were at NHS 111 area code level, this limited what descriptive analyses could be presented for comparisons between the call and online population. The Yorkshire and Humber CUREd10 data were used to compare the population characteristics of those who use online and those who called. However, the CUREd data were from 2016. It was unfortunate that Yorkshire and Humber was not one of the final 18 NHS 111 area codes used in the analyses. Yorkshire and Humber is a large region, which is made up of 22 CCGs, and the NHS 111 online service became live at different times in these CCGs, meaning we could not account for the time point of change in the interrupted times series models. However, this region was used for the descriptive analyses to enable a comparison between the NHS 111 online and telephone populations.

Statistical analysis

Descriptive analyses were used to compare characteristics and the final dispositions for both online and telephone data populations alongside summaries of the characteristics of the online data population for the NHS 111 area codes included in this study.

Interrupted time series

To model the impact introducing the NHS 111 online service may have had on the monthly number of calls, interrupted time series (ITS) was used. However, unlike conventional ITS, a dose–response model was used. This meant instead of modelling the number of calls as a function of the time after the launch of the online service, it was modelled as a function of the number of online contacts that month. The dose–response model allows for the number of online contacts, which may impact on the number of telephone calls, to be taken into account. The dose–response model provided an estimate of the reduction or increase in the number of telephone calls per online contact. Systematic components were also included in the model: an underlying time trend, a step change for when NHS 111 online was introduced and ‘fixed’ seasonal effects (four levels: December–February, March–May, June–August, September–November).

As each NHS 111 area code had different start dates for the introduction of NHS 111 online, each area code was modelled separately and meta-analysis was used to determine the overall effect. Given there were 18 different NHS 111 area codes and a range of outcomes to model, the same model for each site and outcome was used, but different models were used as a sensitivity analysis.

The final model was determined by testing Poisson or Negative Binomial (NB) Generalised Linear Models (GLMs) and whether the model was an Autoregressive (AR) model on four NHS 111 area codes. These four NHS 111 area codes (Hertfordshire, Milton Keynes, North East and Nottinghamshire) were independently chosen prior to any analysis by two statisticians (RMS and RMJ). These sites were chosen as they represented areas with large to small numbers of calls.

To test whether an AR model was appropriate, the primary outcome was differenced to remove the general upwards trend (online supplemental figure 1) and then the AutoCorrelation Function (ACF) and Partial AutoCorrelation Function (PACF) plots were investigated. Following this, it was agreed that for all four area codes, an AR model was not needed but there may be some seasonality. However, it had already been pre-specified that seasonality would be included in the model and would be accounted for with the season variable.

The primary outcome variable was the number (count) of triaged calls to NHS 111 each month so both Poisson and NB models were considered. As the output for the Poisson model showed the data were over dispersed, the NB model was chosen over the Poisson model. Again, the ACF and PACF plots for these models were investigated and it was confirmed an AR model was not needed (online supplemental figures 2, 3).

The final model used for the analysis was:

where the outcome is the number of calls to the NHS 111 telephone service each month, time is a linear variable 0,1,2,…, dose is the number of NHS 111 online contacts for each month, step is a binary variable which is coded 0 before the introduction of NHS 111 online and 1 afterwards and season is a fixed variable that represents the four seasons in the year.

Sensitivity analyses

Two further models were used for sensitivity analyses: an AR(1) model and a non-linear model with a non-linear term for time were used.

These models were applied to all sites and outcomes.

As the Isle of Wight was an NHS 111 area code included in the analysis, due to its size with small call volumes and an atypical urgent care service configuration one further sensitivity analysis was conducted in which the Isle of Wight was excluded.

All three of these models used a log link function. For the linear and non-linear NB model the glm.nb function of the MASS package was used.11 For the AR model the tsglm function of the tscount package was used.12

Meta-analysis

Forest plots were used to summarise the dose from the individual area analyses for all outcomes with estimates displayed as the incidence rate ratio (IRR) per 1000 online contacts. To combine the results from each area, a random-effects meta-analysis was used to estimate the average effect of introducing the online service on each outcome.13 The between-area variance, τ², was estimated using the DerSimonian-Laird method14 and the proportion of total variability due to between-area heterogeneity was evaluated using the I² statistic.15 The results are presented as an overall estimate for each outcome alongside its associated 95% CI and p value. Meta-analysis was conducted using the metagen function of the meta library.16

As above, the meta-analysis was repeated for all sensitivity analyses and the overall estimate and 95% CI for each model was displayed on a forest plot for comparison.

All analyses were conducted using R V.3.6.3 (R Core Team, 2020).17

Patient and public involvement

Patient and public were involved prior to and throughout the wider project. However, there were no patient or public involvement in this part of the study.

Results

Demographics

Table 1 presents the population characteristics for those who contacted NHS 111 via the telephone service (2016) compared with the online service (2019) in Yorkshire and Humber. The largest difference in proportions between the two populations was for age. There was a higher proportion of the younger population using the online service compared with the telephone service with 61.1% of those aged between 16 and 34 using online service compared with 31.2% using the telephone. The results also suggest that the online services have a higher proportion being recommended a 999 ambulance compared with the telephone service, but have a smaller proportion recommended to contact primary care.

Table 1

Characteristics of the NHS 111 telephone and online population for Yorkshire and Humber

Yorkshire and Humber	N Online*	%	N calls †	%
N	275 538	100	1 350 280	100
Sex
Female	186 524	67.7	762 741	56.5
Male	89 014	32.3	585 625	43.4
Not known	–	–	587	0.0
Not specified	–	–	1327	0.1
Total	275 538	100	1 350 280	100
Age
[0,2)	–	–	138 969	10.3
[2,16)	25 636	9.3	188 414	14.0
[16,35)	168 295	61.1	421 536	31.2
[35,75)	78 823	28.6	415 247	30.8
[75+)	2771	1.0	186 096	13.8
NA	13	0.0	18	0.0
Total	275 538	100	1 350 280	100
Time of day‡
Night	95 224	34.6	423 546	31.4
Day	180 314	65.4	926 734	68.6
Total	275 538	100	1 350 280	100
Weekend/week
Week	184 712	67.0	774 167	57.3
Weekend	90 826	33.0	576 113	42.7
Total	275 538	100	1 350 280	100
Disposition
5.23 (Ambulance)	34 571	14.8	135 999	12.8
5.24 (ED)	22 678	9.7	101 840	9.6
5.25 (Primary Care)	176 210	75.5	824 134	77.6
Total	233 459	100	1 061 973	100

*Online data were collected between January–December 2019.

†Telephone data were collected between January–December 2016.

‡Day: 08:00–19:59.

ED, emergency department; NHS, National Health Service.

Online demographics by NHS 111 area code

Table 2 presents the characteristics of those who use NHS 111 online for each NHS 111 area code. The characteristic percentages tended to be similar for each area code. Of those using NHS 111 online, a higher proportion were female, most of the online users were in the younger age categories with very small proportions in the 75+ group, and a larger proportion of contacts were made in the day.

Table 2

Characteristics of the NHS 111 online population split by the 18 NHS 111 area codes

Site	N	Sex			Age						Time of day*			Weekend/week
		Female	Male	Total	[2,16)	[16,35)	[35,75)	[75+)	NA	Total	Night	Day	Total	Week	Weekend	Total
North East	142 373	99 088	43 285	142 373	15 834	81 521	43 239	1775	4	142 373	45 105	97 268	142 373	91 021	51 352	142 373
		69.6%	30.4%	100%	11.1%	57.3%	30.4%	1.2%	0.0%	100%	31.7%	68.3%	100%	63.9%	36.1%	100%
Lincolnshire	26 469	18 302	8167	26 469	2730	15 387	7998	354	–	26 469	9809	16 660	26 469	16 979	9490	26 469
		69.1%	30.9%	100%	10.3%	58.1%	30.2%	1.3%	–	100%	37.1%	62.9%	100%	64.1%	35.9%	100%
Nottinghamshire	36 263	25 169	11 094	36 263	3089	23 047	9761	366	–	36 263	13 473	22 790	36 263	24 143	12 120	36 263
		69.4%	30.6%	100%	8.5%	63.6%	26.9%	1.0%	–	100%	37.2%	62.8%	100%	66.6%	33.4%	100%
Derbyshire	39 085	27 356	11 729	39 085	3770	22 934	11 947	433	1	39 085	14 519	24 566	39 085	25 742	13 343	39 085
		70.0%	30.0%	100%	9.6%	58.7%	30.6%	1.1%	0.0%	100%	37.1%	62.9%	100%	65.9%	34.1%	100%
Isle of Wight	4675	3088	1587	4675	550	2382	1649	92	2	4675	1662	3013	4675	3231	1444	4675
		66.1%	33.9%	100%	11.8%	51.0%	35.3%	2.0%	0.0%	100%	35.6%	64.4%	100%	69.1%	30.9%	100%
Inner North West London	12 955	8247	4708	12 955	524	9084	3255	90	2	12 955	4298	8657	12 955	9514	3441	12 955
		63.7%	36.3%	100%	4.0%	70.1%	25.1%	0.7%	0.0%	100%	33.2%	66.8%	100%	73.4%	26.6%	100%
Hillingdon	6498	4451	2047	6498	567	4020	1848	63	–	6498	2514	3984	6498	4417	2081	6498
		68.5%	31.5%	100%	8.7%	61.9%	28.4%	1.0%	–	100%	38.7%	61.3%	100%	68.0%	32.0%	100%
Hertfordshire	34 320	23 531	10 789	34 320	3607	19 201	11 091	421	–	34 320	12 617	21 703	34 320	22 035	12 285	34 320
		68.6%	31.4%	100%	10.5%	55.9%	32.3%	1.2%	–	100%	36.8%	63.2%	100%	64.2%	35.8%	100%
Cambridgeshire and Peterborough	30 132	20 268	9864	30 132	3047	17 378	9362	344	1	30 132	11 173	18 959	30 132	19 249	10 883	30 132
		67.3%	32.7%	100%	10.1%	57.7%	31.1%	1.1%	0.0%	100%	37.1%	62.9%	100%	63.9%	36.1%	100%
Northamptonshire	26 765	18 398	8367	26 765	2898	15 259	8275	331	2	26 765	9793	16 972	26 765	17 271	9494	26 765
		68.7%	31.3%	100%	10.8%	57.0%	30.9%	1.2%	0.0%	100%	36.6%	63.4%	100%	64.5%	35.5%	100%
Milton Keynes	10 368	7232	3136	10 368	1065	6014	3201	87	1	10 368	3707	6661	10 368	7123	3245	10 368
		69.8%	30.2%	100%	10.3%	58.0%	30.9%	0.8%	0.0%	100%	35.8%	64.2%	100%	68.7%	31.3%	100%
Leicestershire and Rutland	38 235	26 230	12 005	38 235	3740	22 699	11 356	438	2	38 235	13 761	24 474	38 235	25 118	13 117	38 235
		68.6%	31.4%	100%	9.8%	59.4%	29.7%	1.1%	0.0%	100%	36.0%	64.0%	100%	65.7%	34.3%	100%
Outer North West London	20 100	13 488	6612	20 100	1651	12 456	5768	224	1	20 100	7838	12 262	20 100	13 634	6466	20 100
		67.1%	32.9%	100%	8.2%	62.0%	28.7%	1.1%	0.0%	100%	39.0%	61.0%	100%	67.8%	32.2%	100%
North Central London	30 083	20 103	9980	30 083	1942	19 623	8197	321	–	30 083	10 943	19 140	30 083	20 957	9126	30 083
		66.8%	33.2%	100%	6.5%	65.2%	27.2%	1.1%	–	100%	36.4%	63.6%	100%	69.7%	30.3%	100%
South East London	47 243	32 329	14 914	47 243	3357	30 346	13 183	353	4	47 243	18 118	29 125	47 243	32 416	14 827	47 243
		68.4%	31.6%	100%	7.1%	64.2%	27.9%	0.7%	0.0%	100%	38.4%	61.6%	100%	68.6%	31.4%	100%
Bristol, North Somerset and South Gloucestershire	26 046	17 248	8798	26 046	2054	15 830	7903	258	1	26 046	9910	16 136	26 046	17 298	8748	26 046
		66.2%	33.8%	100%	7.9%	60.8%	30.3%	1.0%	0.0%	100%	38.0%	62.0%	100%	66.4%	33.6%	100%
Cornwall	16 786	11 230	5556	16 786	1916	8772	5790	307	1	16 786	6383	10 403	16 786	10 551	6235	16 786
		66.9%	33.1%	100%	11.4%	52.3%	34.5%	1.8%	0%	100%	38.0%	62.0%	100%	62.9%	37.1%	100%
Staffordshire	36 846	25 378	11 468	36 846	4083	21 114	11 225	424	–	36 846	13 588	23 258	36 846	24 189	12 657	36 846
		68.9%	31.1%	100%	11.1%	57.3%	30.5%	1.2%	–	100%	36.9%	63.1%	100%	65.6%	34.4%	100%
Total	585 242	401 136	184 106	585 242	56 424	347 067	175 048	6681	22	585 242	209 211	376 031	585 242	384 888	200 354	585 242
		68.5%	31.5%	100%	9.6%	59.3%	29.9%	1.1%	0.0%	100%	35.8%	64.3%	100%	65.8%	34.2%	100%

*Day: 08:00–19:59.

NHS, National Health Service.

Disposition comparison telephone versus online data

Table 3 presents the dispositions of those who use NHS 111 online versus calls for each NHS 111 area code. The proportion of dispositions for calls versus online are fairly similar.

Table 3

Disposition comparison for NHS 111 telephone and online contacts for the 18 NHS 111 area codes (January–December 2019)

NHS 111 area code	Disposition	Call N	%	Online N	%	NHS 111 area code	Call N	%	Online N	%
North East	5.23 (Ambulance)	128 500	22.2	19 610	16.8	Northamptonshire	28 460	17.9	3934	17.7
	5.24 (ED)	78 262	13.5	14 646	12.5		19 123	12.0	2959	13.3
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	371 186	64.2	82 539	70.7		111 275	70.1	15 395	69.1
	Total	577 948	100	116 795	100		158 858	100	22 288	100
Lincolnshire	5.23 (Ambulance)	28 940	21.6	3901	18.1	Milton Keynes	9005	17.9	1581	19.0
	5.24 (ED)	11 793	8.8	2495	11.6		5163	10.3	1064	12.8
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	93 347	69.6	15 172	70.3		36 175	71.9	5664	68.2
	Total	134 080	100	21 568	100		50 343	100	8309	100
Nottinghamshire	5.23 (Ambulance)	40 777	21.1	5386	18.4	Leicestershire and Rutland	43 298	18.4	5826	18.4
	5.24 (ED)	22 479	11.6	3639	12.5		20 389	8.7	3685	11.7
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	130 182	67.3	20 213	69.1		171 359	72.9	22 122	69.9
	Total	193 438	100	29 238	100		235 046	100	31 633	100
Derbyshire	5.23 (Ambulance)	42 481	18.5	5500	17.1	Outer North West London	29 515	17.8	3062	19.3
	5.24 (ED)	19 818	8.6	3712	11.6		22 197	13.4	1789	11.3
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	167 943	72.9	22 885	71.3		113 947	68.8	11 042	69.5
	Total	230 242	100	32 097	100		165 659	100	15 893	100
Isle of Wight	5.23 (Ambulance)	10 881	18.8	695	19.1	North Central London	37 331	18.2	4188	17.7
	5.24 (ED)	8560	14.8	457	12.5		28 086	13.7	2886	12.2
	5.25 a (Contact primary care) and 5.25b (Speak to primary care)	38 550	66.5	2495	68.4		139 748	68.1	16 587	70.1
	Total	57 991	100	3647	100		205 165	100	23 661	100
Inner North West London	5.23 (Ambulance)	13 345	17.0	1930	19.0	South East London	37 089	12.2	6423	17.0
	5.24 (ED)	10 816	13.8	1292	12.7		36 412	12.0	4497	11.9
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	54 264	69.2	6929	68.3		230 600	75.8	26 972	71.2
	Total	78 425	100	10 151	100		304 101	100	37 892	100
Hillingdon	5.23 (Ambulance)	9387	17.7	1001	18.8	Bristol, North Somerset and South Gloucestershire	39 989	19.8	3495	16.7
	5.24 (ED)	6968	13.2	581	10.9		26 174	13.0	2457	11.8
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	36 630	69.1	3741	70.3		135 925	67.3	14 953	71.5
	Total	52 985	100	5323	100		202 088	100	20 905	100
Hertfordshire	5.23 (Ambulance)	27 582	12.9	5845	20.5	Cornwall	16 904	19.4	2505	18.4
	5.24 (ED)	20 206	9.4	3484	12.2		5997	6.9	1542	11.3
	5.25a (Contact primary care) and 5.25b (Speak to Primary care)	166 821	77.7	19 159	67.3		64 462	73.8	9554	70.2
	Total	214 609	100	28 488	100		87 363	100	13 601	100
Cambridgeshire and Peterborough	5.23 (Ambulance)	30 481	18.5	4002	16.5	Staffordshire	36 875	17.5	5535	18.2
	5.24 (ED)	18 670	11.3	3043	12.6		23 994	11.4	3326	11.0
	5.25a (Contact primary care) and 5.25b (Speak to primary care)	115 909	70.2	17 162	70.9		150 281	71.2	21 503	70.8
	Total	165 060	100	24 207	100		211 150	100	30 364	100

ED, emergency department; NHS, National Health Service.

Triaged calls

Figure 1 presents locally estimated scatterplot smoothing plots for the primary outcome, triaged calls, for the four NHS 111 area codes in which the model was developed. Triaged calls exclude calls that were abandoned and those that were not triaged. The general trend in all four sites is similar, with the number of triaged calls increasing over time.

Figure 1

Locally estimated scatterplot smoothing plots of the number of triaged calls and online contacts for the four test NHS 111 area codes. Blue: Triaged NHS 111 telephone; red: NHS 111 online contacts. NHS, National Health Service.

Figure 2 presents the plots for the ITS model for the four NHS 111 area codes using the primary analysis method (linear Negative Binomial with no AR(1)).

Figure 2

ITS plots for the four test sites. Solid line: ITS model; dashed line: null model (No intervention); solid dots: triaged NHS 111 telephone; hollow dots: NHS 111 online contacts. ITS, interrupted time series; NHS, National Health Service.

The results of the meta-analysis of triaged 111 NHS calls are given in Figure 3. The analysis is for the primary analysis method for each site and then for each sensitivity analysis overall.

Figure 3

Forest plots showing the effect of introducing the NHS 111 online service on the number of triaged calls to the NHS 111 telephone service. (A) Estimated effects for individual areas and the overall average effect from the primary analysis (Negative Binomial GLM). Heterogeneity: I²=71.5% (95% CI: 54.1% to 82.3%). (B) Average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. AR1, Autoregressive 1 model; GLM, Generalised Linear Model; NHS, National Health Service.

Figure 3A shows the forest plot of results for the primary analysis, for each NHS 111 area code and overall. The X-axis is showing the IRR per 1000 online contacts. The overall IRR per 1000 online contacts is 1.013 (95% CI: 0.996 to 1.029, p=0.127). This means that on average for every 1000 online contacts, the number of calls to the NHS 111 telephone service that are triaged has increased by 1.3% (95% CI: −0.4% to 2.9%). However, this result is not statistically significant.

Figure 3B presents the forest plot for the overall results of the main analysis method and various sensitivity analyses. Excluding the Isle of Wight has little effect on the estimate. Including a non-linear term for time has increased the SE and lowered the estimates, but the overall conclusion remains the same. The AR(1) model provides similar incidence rate estimates and CIs.

Total calls

Total calls refer to all calls offered to NHS 111 reflecting how many people attempted to contact the service. Figure 4A shows the forest plot of results for the primary analysis, for each NHS 111 area code and overall. The overall IRR per 1000 online contacts is 1.008 (95% CI: 0.992 to 1.025, p=0.313). This means that on average for every 1000 online contacts, the number of calls to NHS 111 has increased by 0.8% (95% CI: −0.8% to 2.5%). However, this result is not significant.

Figure 4

Forest plots showing the effect of introducing the NHS 111 online service on the total number of calls to the NHS 111 telephone service. (A) Estimated effects for individual areas and the overall average effect from the primary analysis (Negative Binomial GLM). Heterogeneity: I²=68.0% (95% CI: 47.7% to 80.4%). (B) Average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. AR1, Autoregressive 1 model; GLM, Generalised Linear Model; NHS, National Health Service.

Figure 4B presents the forest plot for the overall results of the main analysis method and various sensitivity analyses. Excluding the Isle of Wight has little effect on the estimate. Including a non-linear term for time has increased the SE and decreased the IRR, there is now a 3%–4% decrease in calls per 1000 online contacts, but the overall conclusion remains the same. The AR(1) model provides similar incidence rate estimates and CIs.

Emergency ambulance dispositions

One of the dispositions at the end of a 111 contact is referral to or to call 999 for an emergency ambulance response. The outcome for this analysis is the number of 999 ambulance dispositions for both NHS 111 telephone and online. Figure 5A shows the forest plot of results for the primary analysis, for each NHS 111 area code and overall. The overall IRR per 1000 online contacts is 1.067 (95% CI: 1.035 to 1.100, p<0.001). This means that on average for every 1000 online contacts, the number of recommendations for ambulance response has increased by 6.7% (95% CI: 3.5% to 10.0%). This result is considered a statistically significant effect, suggesting that on average the online 111 service could cause an increase in the number of ambulance dispatches overall if online users follow this advice.

Figure 5

Forest plots showing the effect of introducing the NHS 111 online service on the number of recommendations for ambulance call outs. (A) Estimated effects for individual areas and the overall average effect from the primary analysis (Negative Binomial GLM). Heterogeneity: I²=89.8% (95% CI: 85.4% to 92.8%). (B) Average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. AR1, Autoregressive 1 model; GLM, Generalised Linear Model; NHS, National Health Service.

Figure 5B presents the forest plot for the overall results of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate. The non-linear model also has little effect on the estimate and CIs, the estimates have decreased slightly. Similarly for the AR(1) model.

ED attendances

Another disposition at the end of a 111 contact is recommendation to attend ED. The outcome for this analysis is the number of ED recommendations for both NHS 111 telephone and online. Figure 6A shows the forest plot of results for the primary analysis, for each NHS 111 area code and overall. The overall IRR per 1000 online contacts is 1.050 (95% CI: 1.010 to 1.092, p=0.014). This means that on average for every 1000 online contacts, the number of recommendations to attend has increased by 5% (95% CI: 1.0% to 9.2%). This result is considered a statistically significant effect, suggesting that on average the online 111 service has caused an increase in the number of ED recommendations overall.

Figure 6

Forest plots showing the effect of introducing the NHS 111 online service on the number of recommendations to attend ED. (A) Estimated effects for individual areas and the overall average effect from the primary analysis (Negative Binomial GLM). Heterogeneity: I²=94.4% (95% CI: 92.4% to 95.8%). (B) Average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. AR1, Autoregressive 1 model; ED, emergency department; GLM, Generalised Linear Model; NHS, National Health Service.

Figure 6B presents the forest plot for the overall results of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate. Similarly for the AR(1) model. The non-linear model changes the direction of the effect, however this result is no longer significant (p=0.110).

Contact with primary care

Primary care dispositions at the end of a 111 contact can suggest users either contact or attend different services within different time frames. This includes General Practice (GP) services but also, for example, pharmacy or dentist (community care). The analysis for this section looks at primary care only.

The outcome for this analysis focuses on the number of primary care only recommendations for both NHS 111 telephone and online. Figure 7A shows the forest plot of results for the primary analysis, for each NHS 111 area code and overall. The overall IRR per 1000 online contacts is 1.051 (95% CI: 1.027 to 1.076, p<0.001). This means that on average for every 1000 online contacts, the number of primary care only recommendations has increased by 5.1% (95% CI: 2.7% to 7.6%). This result is considered a statistically significant effect, suggesting that on average the online 111 service has caused an increase in the number primary care only recommendations overall.

Figure 7

Forest plots showing the effect of introducing the NHS 111 online service on the number of recommendations to contact primary care. (A) Estimated effects for individual areas and the overall average effect from the primary analysis (Negative Binomial GLM). Heterogeneity: I²=84.3% (95% CI: 76.4% to 89.5%). (B) Average effects from the primary analysis and sensitivity analyses. Estimates are incident rate ratios per 1000 online contacts. AR1, Autoregressive 1 model; GLM, Generalised Linear Model; NHS, National Health Service.

Figure 7B presents the forest plot for the overall results of the main analysis method and various sensitivity analyses. Again, excluding the Isle of Wight has little effect on the estimate. Similarly for the non-linear model and the AR(1) model. The non-linear model has slightly smaller estimates but is no longer statistically significant (p=0.168).

Discussion

Introducing the NHS 111 online service added another point of access for urgent and emergency care in the NHS. The online service operates in addition to the existing telephone service, not replacing it, hence creating two sources of access. Both these services can direct users to services in the emergency and urgent care system, unless the health problem is suitable for self-care. Interrupted time series analysis was conducted to assess changes in activity following the introduction of NHS 111 online using a dose–response model where the ‘dose’ is the number of contacts with the NHS 111 online service.

The demographic data showed that the largest difference in population characteristics of the telephone and online users was that a larger proportion of younger people used the online service.

The primary outcome was investigating the impact introducing the online service potentially had on the NHS 111 telephone service. The results indicate that overall, the online service had little impact on the number of total and triaged calls, this suggests that the workload for the NHS 111 telephone service may not have increased or decreased since introducing NHS 111 online. This in turn also suggests there has not been a substantial shift to using the online service instead of the telephone service. However, this finding was not consistent as there were four sites that showed a reduction in triaged calls. This could indicate for these areas that there may have been a shift away from the telephone service to the online service.

For the secondary outcomes which looked at the wider NHS urgent care system, the results from the combined activity from the NHS 111 telephone service and online service suggested that there was an increase in the overall number of recommendations to contact or attend those services following the introduction of the NHS 111 online service. On the surface, the results suggest there was an overall increase in demand for emergency and urgent care services, which is not surprising. For the 18 sites we included in our analyses, there were almost 600 000 contacts with the NHS 111 online service with no visible shift away from the telephone service and nationally there were over 2 million contacts during 2019. It has been shown previously, that introducing new services and access points for emergency and urgent care, such as NHS Direct, NHS 111 and Walk in Centres, have created an increase and therefore new demand for services.18–20 Following from this, it is entirely plausible that introducing this new online service could produce the same effect. However, the findings from the previous research were based on actual utilisation of other services in the emergency and urgent care system. For this analysis, it was only possible to show the recommendations about services to contact or attend and so potential increases in service utilisation. This estimated potential service use increase would only hold true if all recommendations were acted on and if those who used NHS 111 online subsequently accessed a service they would not have used without a recommendation from the online service.

There is also the possibility that any changes in demand may have been influenced by other external factors. Meta-analysis was used to produce an overall summary measure of effect from the 18 sites included in the analyses. However, the forest plots show there is considerable variation between different NHS 111 area codes, this could suggest there are local differences, for example service availability and the amount of integration between services, therefore the effect of introducing NHS 111 online maybe inconsistent in different health economies.

Strengths and limitations

There are a few limitations of these analyses to be discussed. First, as NHS 111 online was rolled out rapidly as a national service, it was not possible to use an experimental design with control area codes. This means any effects seen cannot be assumed to be the direct result of introducing the new service or whether they would have happened anyway due to other factors impacting on both the NHS 111 telephone service and the wider emergency and urgent care system.

Second, as we had to use the telephone service NHS 111 minimum dataset aggregated data rather than patient-level data, this meant we were only able to successfully match 18 of the 38 potential NHS 111 area codes to NHS 111 online data, therefore, we have not been able to establish a national estimate of impact. However, for the 18 NHS 111 area codes included in the analysis we are confident that they are representative of different geographical areas, activity volume and provider types across England to make reasonable inferences.

Third, as this evaluation of NHS 111 online was conducted during the early stage of implementation, it had only been operational for 12–18 months in the sites we have used, we have estimated system impact based on the ‘dose’, in terms of contacts with the new service, present at that time. Analysing the data at a later stage when the service becomes more widely understood by the public, contacts may increase and it is possible the impact may change, therefore any subsequent assessment of impact could be more robust.

Finally, as previously discussed, the data and results only consider recommendations for care and not actual care received. This might be quite different and will be dependent on how people use the service making it difficult to estimate how much new demand there may be.

Further work

The work from the study has opened up a number of potential areas to conduct further work. Exploring the patient-level comparisons further of the characteristics of the two NHS 111 populations (Telephone and Online) and the relationships between characteristics. This has the potential to help identify patients who are most likely to benefit from using the two types of service and provide information that would help patients choose which service to use.

As discussed in the limitations, we were only able to determine recommendations to other services, not what happened. Further work is required to explore the dispositions further to determine whether NHS 111 has had an impact on the wider services. To do this linked data would be required to follow the patient pathway and unfortunately, this linkage is currently not possible as NHS 111 online have no individual patient identifiers. An example of using NHS 111 telephone linked data is the work by Lewis et al.21

In the results, there was evidence of inconsistencies between different NHS 111 area codes, some seeing decreases and others seeing increases, further work could explore whether the differences in impact on the 111 telephone service between areas are due to different populations, available services, policies or other factors.

Evaluating the services early in the introduction does have the potential to have unstable results. Repeating the analysis at a later stage once the systems have settled and matured and the population are more familiar with the purpose and use would help provide a more clear picture. Analysing the data at a later date could also provide the opportunity to include all the NHS 111 area codes as the introduction stage could be ignored in the analysis. This could be even more important now following the COVID-19 pandemic, where the NHS 111 systems saw huge increases in demand for the telephone service and the NHS 111 online service was rapidly developed to include a COVID-19-specific triage pathway to help deal with the demand. With the publicity the NHS 111 telephone and online services received during the start of the pandemic, it would be of interest to see whether population behaviour of using these services has changed since the pandemic started.

Conclusion

The results show that younger people are more likely to use NHS 111 online compared with older people. It was also found that the NHS 111 online service has little impact on the number of triaged and total calls, suggesting that the workload for NHS 111 has not increased or decreased as a result of introducing NHS 111 online. There was evidence that the introduction of NHS 111 online increased the overall number of disposition recommendations (ambulance, ED and primary care) of the NHS 111 telephone and NHS online services combined. However, as these are recommendations it is not possible to say whether this will have increased the workload for the rest of the urgent care system services. It will be important to further monitor impact as contacts with the NHS 111 online service increase and avoid creating large volumes of new demand in a system that is already under serious pressure.