Acceptability of NHS 111 the telephone service for urgent health care: cross sectional postal survey of users' views.

BACKGROUND: In 2010, a new telephone service, NHS 111, was piloted to improve access to urgent care in England. A unique feature is the use of non-clinical call takers who triage calls with computerized decision support and have access to clinical advisors when necessary. Aim. To explore users' acceptability of NHS 111.
DESIGN: Cross-sectional postal survey.
SETTING: Four pilot sites in England.
METHOD: A postal survey of recent users of NHS 111.
RESULTS: The response rate was 41% (1769/4265), with 49% offering written comments (872/1769). Sixty-five percent indicated the advice given had been very helpful and 28% quite helpful. The majority of respondents (86%) indicated that they fully complied with advice. Seventy-three percent was very satisfied and 19% quite satisfied with the service overall. Users were less satisfied with the relevance of questions asked, and the accuracy and appropriateness of advice given, than with other aspects of the service. Users who were autorouted to NHS 111 from services such as GP out-of-hours services were less satisfied than direct callers.
CONCLUSION: In pilot services in the first year of operation, NHS 111 appeared to be acceptable to the majority of users. Acceptability could be improved by reassessing the necessity of triage questions used and auditing the accuracy and appropriateness of advice given. User acceptability should be viewed in the context of findings from the wider evaluation, which identified that the NHS 111 pilot services did not improve access to urgent care and indeed increased the use of emergency ambulance services.

Introduction

In 2010, a new telephone service NHS 111 was launched in four pilot areas in England. The aims of the service were to improve access to and satisfaction with urgent care, make more efficient use of emergency and urgent care services by directing people to the most appropriate service, and in the longer term reduce use of emergency ambulances (1). Members of the general population called a free, easy to remember number, ‘111’, and spoke to a trained non-clinical call taker who used the software NHS Pathways to direct people to the most appropriate health care. Call takers had access to clinician support if needed, usually a nurse. The service also took telephone calls on behalf of general practice (GP) services operating outside the normal working hours of Monday to Friday during the day (‘GP out-of-hours services’). Calls to GP out-of-hours services were automatically put through to NHS 111, or a message on GP answer phones directed people to call NHS 111 out-of-hours. The new service was an addition to a set of services meeting emergency and urgent care needs in England including emergency ambulances, emergency departments, urgent care centres, GP out-of-hours services and daytime GP. The four pilots operated while NHS Direct, a 24-hour nurse-led telephone service for health advice, was part of the emergency and urgent care system, although the national policy plan was to replace the telephone service provided by NHS Direct with NHS 111 during 2013.

The NHS 111 pilot service was established in response to confusion in the general population about how to access services for urgent care (2,3). The rationale was that NHS 111, with an emphasis on offering people an easy and fast response to an urgent situation, could answer the telephone immediately and direct people to the ‘right service, first time’. It is important to identify users’ views of this service because the consequences of poor acceptability may be lack of adherence to advice or treatment, complaints and litigation or avoidance of future use. The aim of this study was to explore users’ views of NHS 111 in four pilot sites prior to national roll out of the service to highlight any improvements needed for the national service, as well as provide evidence about future international models of telephone access to care using non-clinical call takers.

Method

We undertook a cross-sectional postal survey of recent users of NHS 111 in four pilot sites between July and October 2011, around 9–11 months after the start of the service. We planned to send questionnaires to 1200 users in each site. Calls made in a single week were used as the sampling frame for sites with a large number of calls. A 2-week sampling period was used for sites with lower numbers of calls. When >1200 calls were identified in a week, systematic random sampling was used to select 1200 calls. After sampling, a small number of calls were excluded by NHS 111 staff: patients aged ≤15 years if algorithms related to sexual health used because the questionnaire was addressed to the parent/guardian of this age group; users who had not provided their home address details and the second or more calls of repeat callers in that time period.

Personnel at each site sent a covering letter, information booklet, questionnaire and reply-paid envelope to the patient within 3 weeks of the call. In most cases, the caller and the patient were the same person. Where calls were made on behalf of children, we addressed the questionnaire to ‘care of the parent/guardian of’. We asked in the covering letter that both caller and patient attempt to complete the questionnaire together if relevant and possible. Responses were returned directly to our team. Questionnaires had unique identifiers and sites were informed of which users needed reminders. Up to two reminders were sent to non-responders ~3 weeks and 6 weeks after the initial mailing.

The questionnaire

We asked how people had accessed the service and the advice they were given. Then we addressed the multidimensional aspects of satisfaction (4), asking about patient-centred care (helpfulness of staff, reassurance), access (clarity about when to use the service), communication and information (relevance of questions asked), technical quality (whether the advice worked well in practice) and efficiency (speed with which problem as dealt with). Most of the questions about satisfaction had response sets of 5-point Likert scales from ‘strongly agree’ to ‘strongly disagree’. We included a question about overall satisfaction, on a 5-point Likert scale from ‘very satisfied’ to ‘very dissatisfied’. We also addressed two potential consequences of poor acceptability: compliance with advice and views of whether they would return to the service with a similar health problem. We measured caller demographics. Because these types of surveys have been criticized for ceiling effects (4), we added an open question to identify aspects of dissatisfaction as well as satisfaction (5). We gave respondents a short section to describe in their own words anything with which they were particularly satisfied or dissatisfied. The questionnaire was similar to one used in a previous evaluation of telephone delivered health care by NHS Direct (6). We used similar questions to allow comparison between the findings from the evaluations of pilots of NHS 111 and NHS Direct. We undertook a small pilot with three NHS 111 users in one site who discussed within a telephone interview the face and content validity of the questionnaire. We were aware before we embarked on this survey that NHS 111 users might not know that they had called the service because some users were autorouted from other services such as GP out of hours. We designed the covering letter and questionnaire to address this.

Analysis

Data were analysed in PASW Statistics version 18. Not everyone answered all questions. Missing values were excluded, so we report the denominator for all results. We described the views of respondents, reporting 95% confidence intervals for key statistics. We undertook further analysis on the set of questions using the response set ‘strongly agree’ to ‘strongly disagree, using the Friedman test to identify whether satisfaction differed by aspect of the service. Then we dichotomized the satisfaction items at ‘strongly agree’ versus other categories because people who use the extreme category of a satisfaction variable tend to have no negative comments to make, whereas people who use any other category can identify issues they were unhappy about (7). We undertook a logistic regression to test whether satisfaction differed by route into the service, adjusting for age group (16–44, 45–64, 65+), gender (male, female) and ethnicity (white, ethnic minority group). We analysed the open question using a ‘quantitative strategy’ similar to content analysis (8). One researcher (EK) read the comments and identified a thematic framework developed inductively from the comments. EK then applied the coding frame to each comment, counting the numbers of respondents making different types of comments.

Results

The response rate was 41% (1769/4265), with 49% (872/1769) of respondents making written comments. The majority of callers were female, younger than 65 years and white (Table 1). The median age was 45 and interquartile range 32–63.

Table 1.

Respondent demographics

Characteristic	%	n/N
Female	71	1237/1750
Age
16–44	47	832/1754
45–64	28	483/1754
>65	25	439/1754
White	85	1479/1736
Disability or long-term illness	34	577/1718
Home owners	60	969/1610

Accessing NHS 111

Two-thirds of respondents had dialled NHS 111 directly, with others ‘autorouted’ from another service or directed from another service (Table 2). Eighty-six percent (1495/1731, 95% confidence interval: 85% to 88%) reported being ‘definitely clear’ about when to call NHS 111.

Table 2.

Reported route into NHS 111 and advice given by NHS 111

	%	n/N
Route into NHS 111: dialled NHS 111 directly	62	1059/1712
Called another service such as general practice out of hours and put through automatically to NHS 111 (‘autorouted’)	24	416/1712
Called another service, usually their own general practice out of hours, and heard a message directing them to call NHS 111	11	193/1712
Unsure	3	44/1712
Reported advice given: NHS 111 arranged an appointment at a centre such as an urgent care centre	34	578/1690
Transferred to the ambulance service or sent an ambulance	20	346/1690
Told to visit a centre such as an emergency department, walk in centre or urgent care	15	256/1690
Contact their own general practice	9	154/1690
Self-care	10	167/1690
Other (e.g. a health visitor, dentist, district nurse)	11	189/1690

Advice given

A fifth of respondents reported being transferred to the ambulance service or sent an ambulance, and a tenth reported advice to self-care only (Table 2). Respondents were asked how helpful they found the advice given by NHS 111. Overall, two-thirds of respondents reported receiving ‘very helpful’ advice from NHS 111 (65% 1108/1695, 95% confidence interval: 63% to 68%). Respondents were more likely to feel that the advice given was very helpful if this advice directed the patient to the ambulance service (76%) or if an appointment was arranged for the patient (71%) than if they were asked to self-care (64%), visit a health centre (55%), contact an ‘other’ service (54%) or contact their own GP (52%; P = 0.001).

Satisfaction with different aspects of NHS 111

Respondents were asked a closed question about their overall satisfaction with NHS 111. Seventy-three percent (1255/1726, 95% confidence interval: 71% to 75%) were very satisfied with the way NHS 111 handled the whole process, 19% (319/1726) were fairly satisfied and 5% (79/1726) were dissatisfied. Eight hundred and seventy-two respondents provided written comments, of whom 867 had also completed the closed satisfaction question. Seven percent (63/867) of people completing the open question reported that overall they were dissatisfied with NHS 111 compared with 2% (16) of the 859 who did not complete the open question. That is, more people who were dissatisfied took the opportunity to write comments than those who were satisfied.

Respondents were asked to ‘strongly agree’ through to ‘strongly disagree’ on a 5-point Likert scale with a series of positive statements about NHS 111 (Table 3). Small percentages of respondents disagreed or strongly disagreed with these statements. Some statements had different distributions of responses than others (P = 0.001). Two aspects of the service were less acceptable than others based on these closed questions and the responses to the open question (Table 3). There were concerns about the relevance of questions asked and whether the advice given worked in practice. These two statements had the lowest proportion of respondents strongly agreeing. The themes developed inductively from the open comments mapped onto the aspects of the service addressed in the closed questions. More people chose to write positive comments about the service than negative comments except for the issue of relevance of questions asked. Indeed, 8% of all written comments were negative comments about the relevance of questions asked, showing the relative dissatisfaction with this aspect of the service. (Table 3). The next largest set of negative comments related to dissatisfaction with the advice given because people did not think that they had been directed to the right service.

Table 3.

Satisfaction with different aspects of the NHS 111 service

Statement	Strongly agree	Agree	Neither	Disagree	Strongly disagree	N = 100%	Positive comments*	Negative comments*
The 111 staff were helpful	63%	30%	4%	1%	1%	1725	N = 245 ‘They were very caring, friendly and reassuring’	N = 14 ‘Never mentioned if she gets worse ring all he said ok bye wait for their call - how rude’
The questions asked by the 111 service were relevant	50%	36%	8%	5%	2%	1688	N = 38 ‘Satisfied: questions asked were relevant and in detail assuring me that I was talking to the right person’	N = 78 ‘The checklist the operator read was irrelevant and she apologised for this’
The 111 service dealt with my problem quickly	58%	31%	6%	3%	2%	1702	N = 172 ‘They didn’t take ages to help me, they assessed what my problem was and got through the emergency questions quickly efficiently to get on with what was wrong’	N = 27 ‘It took what seemed like a long time to discuss with first person on the line, to then be passed on to someone more qualified, to then have to discuss again when all I wanted was an appointment with the out of hours service’
The advice I was given by the 111 service worked well in practice	50%	35%	9%	3%	2%	1651	N = 118 ‘I felt very reassured that I was being given good advice and steps to follow’	N = 59 ‘Was advised the walk in centre could deal with my issue, only to then be told by the walk in centre they couldn’t help’
The 111 service helped me to make contact with the right health service	53%	32%	9%	3%	2%	1605	N = 100 ‘I had the help I needed in a matter of minutes, an ambulance came so very quick and a paramedic came in a car before that’	N = 42 ‘After waiting 3 hours for a doctor to return my call we went back to the walk in centre where we were told 111 had gotten the wrong phone number despite being given it 3 times’
Using the 111 service reassured me	55%	30%	9%	4%	3%	1679	N = 138 ‘The service that 111 give is exceptional. He [NHS 111 advisor] calmed me down and reassured me everything was going to be ok’	N = 3 ‘The service did not reassure me and I was later admitted to hospital as an emergency’
I was completely happy with the 111 service	59%	28%	7%	4%	3%	1706	‘Brilliant service, start to finish’	‘I think it [NHS 111] is a complete waste of NHS resources and yet another hair brained scheme by the government’
The 111 service is a valuable addition to the NHS	65%	24%	6%	2%	3%	1711	–	–

Compliance with advice and willingness to use service again

The majority of respondents reported complying with all the advice given by NHS 111 (86%, 1435/ 1670, 95% confidence interval: 84% to 88%). A similar proportion reported being willing to use the service again for a similar health problem (86%, 1490/1729).

One in 10 complied with some of the advice (11%, 180/1670) and only a few people reported that they had not followed any of the advice given (3%, 55/1670). Respondents who did not fully comply with the advice given by NHS 111 (n = 235) were asked to indicate their main reason for not following the advice. Most respondents provided a reason for not complying (70%, 165/235), reporting that they did not agree with the advice (21%, 35/165), felt unable to follow it (20%, 30/165), felt that it did not work (19%, 32/165), felt that the health problem changed (7%, 11/165) or that they did not understand the advice (2%, 4/165). A third of people ticked ‘other reason’ that consisted of a diverse range of responses including ‘I forgot’ or ‘it was too late’ (30%, 50/165). While reported compliance was high, it differed by type of advice given. Respondents reported being more likely to fully comply with advice regarding the ambulance service (92%) or if an appointment had been arranged for the patient (91%) than other types of advice: self care (85%), visit a health centre (83%), contact own GP (78%), contact an ‘other service’ (74%; P = 0.001).

Differences by route into NHS 111

The NHS 111 call handling services in the four pilot sites were operated by different providers (one by an ambulance service and three by NHS Direct). There were differences in users’ views between sites. We tested for differences taking age, gender and ethnicity of users into consideration because these can affect users’ views. We also took the route into NHS 111 into consideration because some sites had higher proportions of direct users than others. We found that users’ views differed by the route into NHS 111 (Table 4). Respondents reporting that they had been autorouted to NHS 111 from another health service such as GP out of hours were less satisfied than those who had dialled ‘111’ or received a health service telephone message to dial ‘111’, even after adjusting for demographic characteristics. Once demographics and route into NHS 111 was adjusted for, there was only one statistically significant difference between sites. A higher proportion of users in two sites were clearer about when to use the service (90% and 91% compared with 77% and 83% in other sites, adjusted P = 0.014).

Table 4.

Percent of respondents ‘strongly agreeing’ with satisfaction statements by route into NHS 111

Statement	Direct dial % (n)	Autorouted % (n)	Message to redial % (n)	Adjusted P valuea
The 111 staff were helpful	68 (713)	53 (212)	67 (126)	0.000
The questions asked by the 111 service were relevant	53 (539)	47 (184)	52 (96)	0.139
The 111 service dealt with my problem quickly	62 (640)	49 (194)	62 (114)	0.001
The advice I was given by the 111 service worked well in practice	54 (541)	42 (159)	52 (93)	0.001
The 111 service helped me to make contact with the right health service	57 (557)	45 (164)	57 (102)	0.001
Using the 111 service reassured me	59 (598)	48 (187)	56 (102)	0.004
I was completely happy with the 111 service	63 (649)	50 (197)	62 (117)	0.001
The 111 service is a valuable addition to the NHS	70 (721)	57 (229)	65 (123)	0.001
Overall satisfaction (v satisfied)	76 (791)	64 (259)	76 (144)	0.001
N	1035	408	189	–

aAdjusted for age group, sex and ethnicity.

Discussion

Summary and comparison with existing literature

Overall satisfaction with NHS 111 was very good, with 73% of respondents reporting that they were very satisfied with the new service and 91% that they were very satisfied or satisfied. Caution is required when making comparisons with other services because satisfaction is often measured and reported in different ways. Given this limitation, satisfaction appeared to compare well with satisfaction with telephone consultations in GP settings: 98% (satisfied or very satisfied) (9), 88% (very or fairly satisfied) (10) and 62% (very satisfied) (11). It also compared well with international evidence on general population based telephone triage services at 89% (satisfaction score of 7 or above out of 10) (12) and 73% (very satisfied) (13). One of our pilots had offered a similar service prior to operating NHS 111, and evaluators of that service found very high levels of user satisfaction (14).

The percentage of users finding the advice very helpful was 65% compared with 76% of users of the NHS Direct pilots in their first year of operation (6), even though the age distribution of NHS 111 survey respondents was older than that of NHS Direct survey respondents: median 45 (interquartile range 32–63) versus median 35 (interquartile range 29–47) (6) and older people are usually more satisfied with health care (15). That is, early users of NHS 111 reported finding it less helpful than early users of NHS Direct. The differences in satisfaction and indeed the differences in age distribution between NHS 111 and NHS Direct at pilot stage may be explained by differences in the source of users: NHS 111 took calls for GP out-of-hours services while NHS Direct did not do this at pilot stage.

The majority of our respondents (86%) indicated that they complied with all of the advice given, comparing well with compliance rates for other telephone triage services which ranged from 56% to 98% (median 77%) (16). A surprising finding was that 86% of users reported being ‘definitely clear’ when to call NHS 111 even though this was a new service and only 62% had actually dialled 111 directly. This may be related to the extensive use of local publicity about when to use the service in some of the sites (17).

Two aspects of the service had lower levels of satisfaction—the relevance of questions asked and the advice given. Both of these concerns were also expressed by users of a similar service in one of the pilot sites (14). There was no evidence that the acceptability of the service differed by site. However, there was some evidence that people who call their GP out-of-hours service and expect to speak directly to them were less satisfied than people who chose to call NHS 111. There was no evidence that users were concerned about non-clinical call takers.

Strengths and limitations

This was a large survey of users of NHS 111 and provides the first evidence of satisfaction with this new service and with telephone triage by non-clinical call takers. The response rate of 41% was low. It is comparable with other large postal surveys of access to GP in England, which obtained response rates of 41% in 2008 and 38% in 2009 (18), and a telephone survey of a service similar to NHS 111 with a response rate of 30% (14). Nonetheless, it is likely to suffer from non-response bias. We were unable to estimate non-response bias because we collected demographic information on callers completing our questionnaire and NHS 111 collects demographic information on patients. However, we do know that our respondents were more likely to report being given advice to use the ambulance service than NHS 111 routine statistics reported for the same time period (20% in our survey versus 11% in routine data) and less likely to report self-care only (10% in our survey versus 30% in routine data). Given that respondents with ambulance referrals found NHS 111 more helpful than those asked to self-care, this survey is likely to overestimate the percentage of positive views of NHS 111, although in practice this is a small overestimate and remains within levels found in the existing literature on other similar services. Two other issues suggest that our survey may give a more positive view of acceptability than might exist amongst users. First, satisfaction surveys are known for reporting high levels of satisfaction, with concerns that they may not uncover dissatisfaction, or that low expectations may be the cause of high satisfaction (4,5). Second, some of the questions we used were positively worded and this may have resulted in positive findings. Because of these problems, we also included an open question to allow people to express positive and negative views and indeed these highlighted problems with relevance of questions and advice given. Finally, this report of users’ acceptability must be considered alongside other aspects of the new service (17). The wider evaluation found that in its first year of operation in four pilot sites NHS 111 did not deliver the expected benefits of improving access to urgent care or shifting the care of patients to urgent rather than emergency care. Indeed, it increased the use of emergency ambulance services (17,19).

Implications

Satisfaction rates with the pilots of this new service in their first year were high and many users wrote positive comments about it. This must be interpreted in the light of a ceiling effect when measuring patient views and the probability that non-response bias resulted in an over estimate of positive views. We used open questions to ensure we could identify aspects of dissatisfaction and highlighted two areas for improvement for the service to address. Some users had a problem with the relevance of questions asked. Service providers may argue that some questions need to be asked to minimize the risks of missing health problems and that a balance is required between safe operation of a service and user acceptability. It is likely that there will need to be some trade-off between clinical requirements for safe practice and user satisfaction. However, our view is that the frustration expressed by some users in this survey highlights a need to revisit the content of algorithms used within the software. An expert panel assessing call recordings, undertaken in our wider evaluation, also identified seemingly irrelevant questions and suggested that this might affect some groups and conditions more than others (17). Some of our respondents also expressed concerns about the accuracy and appropriateness of advice given. It may be that the most appropriate advice from a clinical viewpoint is not the advice people wish to hear. However, again, the expert panel in our wider evaluation identified some cases where the advice seemed inappropriate (17), suggesting that refinement of some algorithm endpoints may be needed.

The two issues that caused frustration to users may be related to the software used or to the use of non-clinical call takers for triage. The clinical assessment software is a core ingredient of NHS 111 and we would suggest that the first step to improving the service would be consideration of adjustment of some of the questions and algorithms within the software where particular problems have been identified. It may also be the case that a more cautious approach to decision-making is required when using non-clinical rather than clinical call takers, or they may have less leeway than clinicians to override advice recommended by the software, although research suggests they operate in a similar way to clinical call takers (20). What is interesting is that the open comments on our survey did not show positive or negative reaction to non-clinical call takers. It may be that people are used to this because call takers for the emergency ambulance service are non-clinical, that users obtained clinical advice when they needed it or that the dissatisfaction with relevance of questions asked within NHS 111 was partly due to the use of non-clinical call takers.

There will always be some dissatisfaction with any service and it is important to consider whether change to NHS 111 is necessary. A core aspect of NHS 111—the clinical assessment software—may not be as accommodating of the diverse range of individual situations faced by users as it could be. It is important to address this because patients in the current health system in the UK have options other than NHS 111 and can select them in the face of frustration with the service, e.g. call 999 or attendance at an emergency department.

Declaration

Funding: This is an independent report commissioned and funded by the Policy Research Programme in the Department of Health. The views expressed are not necessarily those of the Department. Reference Number 0490016.

Ethical approval: the study was approved as part of a wider evaluation by Leeds (Central) Research Ethics Committee 10/H1313/57.

Conflict of interest: Professor Alicia O’Cathain and Professor Jon Nicholl are co-applicants on an NIHR Applied Research Programme led by Professor Chris Salisbury on behalf of NHS Direct: the Healthlines study. The focus of this project is the evaluation of NHS Direct delivering telehealth interventions for long-term conditions. NHS Direct staff are part of this research team. In June 2012 a family member of Professor Alicia O’Cathain won a contract to offer patient feedback for NHS 111 sites in London.