Use and usability of GP online services: a mixed-methods sequential study, before and during the COVID-19 pandemic, based on qualitative interviews, analysis of routine eConsult usage and feedback data, and assessment of GP websites in Devon and Cornwall, England.

OBJECTIVES: We explored use and usability of general practitioner (GP) online services.
SETTING: Devon and Cornwall, England.
DESIGN: Mixed-methods sequential study based on qualitative interviews, analysis of routine eConsult usage and feedback data, and assessment of GP websites.
METHODS: First, we interviewed 32 staff and 18 patients from seven practices in June 2018. Second, we used routinely collected consultation meta-data and, third, patient feedback data for all practices using eConsult from June 2018 to March 2021. Lastly, we examined GP websites' usability in January 2020 and September 2021.
RESULTS: Interviews suggested practices infrequently involved patients in eConsult implementation. Some patients 'gamed' the system to achieve what they wanted. Usage data showed a major increase in eConsult resulting from COVID-19. Women used eConsult twice as much as men. Older had similar eConsult consultation rates to younger patients. Patient feedback forms were completed for fewer than 3% of consultations. Patients were mostly satisfied with eConsult but some had concerns about its length and repetitiveness, lack of continuity over time and between eConsult and medical records. We did not find clear evidence that patients' suggested improvements were acted on. Finally, few GP websites met accessibility guidelines and may hinder access to online national services such as eConsult.
CONCLUSION: Given that, face to face, older people consult more, usage data suggest that older people have reduced online access. That the female-to-male ratio of eConsult use use was even greater than 'traditional' face-to-face ratio was unexpected and needs further research. Although eConsult collects and uses routine patient feedback to improve the system, more open systems for patient feedback, such as Care Opinion, may be more effective in helping online systems evolve. Lastly, we question the need for GP websites and suggest that national or regional services are better placed to maintain accessible services. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Our study is a comprehensive exploration of usability using four datasets, including 3 years of eConsult routine data, for Devon and Cornwall.

The analyses were done by independent researchers with no conflicts of interest.

One potential limitation is that a pragmatic approach (following the data) was used, rather than preplanned hypothesis-testing study.

Additionally, routine data are for episodes rather than individuals; some patients and their feedback may be repeated.

The low response rate for the routinely collected feedback data may mean these data are not representative of all patients using eConsult.

Introduction

Various forms of non-face-to-face contact are used in general practice including online triage, videoconsultation and email.1 2 In the UK, the most frequently used primary care triage system is eConsult (previously known as WebGP).3 eConsult is designed to provide patients with an alternative way of contacting their general practitioner (GP) practice.4 It functions not as an online consultation, but as online contact with the practice with subsequent interactions by phone or face to face. In use since 2014, eConsult was used by 1186 practices in the months leading up to the pandemic, with a total population of approximately 20 million patients (personal communication, David Evans eConsult, 2019).

The pandemic accelerated the uptake of such systems.5 Although there is debate about how much we will revert to face-to-face consultation when the pandemic ends, such digital methods are likely to remain, but evolve.6 One concern raised about all digital health systems is accessibility.7–9 Accessibility is determined partly by personal eHealth readiness8 dependent on online skills, physical access to the internet, support from digital carers or champions and cost. However, accessibility is also determined by the usability of the online service (its interface and user experience). Frequency of use is one indicator of accessibility. Patient feedback and formal usability assessments are other ways to judge accessibility. Like other technologies, online patient services like eConsult, evolve as interfaces improve, as costs reduce, as skills improve, and as their use is built into everyday practice and they become ‘normalised’.10 This study considers how we ensure that eHealth systems improve and evolve keeping accessibility and usability as key aims.

Health lags sectors such as banking and tourism in the transition to greater use of online. More than any other sector equity of access is important. As the banking sector moved much of its activity from the high street to online, there was appropriate debate about those who were disadvantaged.11 Concerns remain and some banks (eg, Barclays Digital Eagles) set up national systems of support to help customers.12 Concerns about accessibility are also partly addressed by improvements in the usability of banking software but even these well-resourced websites sometimes fall short on accessibility.13 14 As technology becomes more usable so it becomes accessible to more people but accessibility for subgroups may change during this transition. When banks were open on the high street they were very accessible to retired people who were free during the day but difficult to access for younger people who were working full time. When they moved online this reversed, younger digitally literate people then had access 24/7 while some digitally excluded people lost or had diminished access.

To improve accessibility, many digital health systems routinely try to capture patient feedback but it is not clear how much this is used by providers or how useful it is. Often studying use can reveal who is (and therefore who is not) using systems and for what.

As one example of an eHealth system in general practice, we aimed to explore the use and usability of the system eConsult. At the time of study most patients accessed eConsult via their GP website with the eConsult home page offering three options (1) help with a condition, (2) general advice or (3) administrative help. Some practices offered a fourth option, (4) help for a child. eConsults submitted by patients were sent by the system to the practice where they were reviewed by practice staff. Users were told that their request would be responded to within 48 hours.

Our study started prepandemic in 2018 when we were exploring the, at the time, relatively slow uptake of eConsult by practices in Devon and Cornwall (D&C). The pandemic meant that use of eConsult both by practices and patients increased rapidly in 2020. Monitoring of routinely collected data across this period allowed us to compare usage by patients between sub groups by demography and practice, and to assess patient feedback on the system. This study started just as an exploration of eConsult. However, the variation in rates of use by patients between practices in the early part of the study led to further examination of GP websites as being one possible reason for these differences. The aim of this study, therefore, was to explore the use and usability of eConsult and GP websites using four datasets.

Methods

Context and timing

D&C have an older population (26% of adults are aged 65+ compared with to 21%15 for England) and a rural population with poor transport links in many places. In June 2018 there were three clinical commissioning groups (CCGs) in the counties of D&C that bought primary care services from a total 33016 general practices (including branches) in Devon (206) and Cornwall (124). Thirteen practices, mostly in Devon, were already using eConsult (earliest start from January 2017). The three CCGs offered eConsult licences to all practices in the autumn of 2017. In April 2019, the South Devon and North East and West Devon CCGs merged to become the Devon CCG. In February 2020, the COVID-19 pandemic led to restrictions on face-to-face GP appointments and a major uptake in the use of online triage systems such as eConsult.

Design

This study was a mixed-methods sequential design, bringing together four datasets from three sources. We started in 2018 with qualitative implementation interviews pre-COVID. Second we analysed two independent datasets -routine consultation episode data and patient feedback data—collected by eConsult for 33 months (June 2018–March 2021) before and during the pandemic. Lastly, large differences in early uptake of eConsult by practices led to exploration of usability of Cornish GP websites. We draw on findings from these four datasets to consider the evolution of accessibility and usability of online services.

Implementation interviews regarding eConsult

From the 95 practices using eConsult in June 2018 we recruited seven practices. This was a purposive sample giving a range of geography (5 Devon, 2 Cornwall), list size (4750–13000) and length of use (3 started 2017, 4 in 2018). Within the seven practices an opportunistic sample of 50 stakeholders (11 clinical staff, 21 administrative staff and 18 patients) were interviewed, a pragmatic sample size and distribution to capture a range of views. We aimed to interview 30–40 people, mirroring the sample size used in a previous qualitative evaluation of eConsult in 2018.4 Participants were recruited through email and telephone conversations with GP practice staff aiming for good representation of patients, clinical and administrative staff. Semistructured interviews were audiorecorded and transcribed verbatim. Interview schedule (online supplemental appendix) and data analysis were informed by normalisation process theory (NPT).17 18 The NPT theoretical framework comprised four domains exploring how stakeholders (1) understand, (2) engage with, (3) work with and (4) appraise the value of a digital health innovation, in this instance eConsult. Interview schedules were reviewed for use by an ethics committee and two university researchers. For this paper, we have focused on sub-themes related to usability.

Analysis of routine eConsult data

Routinely collected data was provided by eConsult for all practices in D&C that had used eConsult from June 2018 to March 2021. This comprised (1) data on eConsults and (2) routine feedback from patient questionnaires including free text comments. Both consultation meta-data (ie, simple details such as practice, date, time and patient demographics) and feedback data were episodic and did not identify individual patients and therefore patients could have been included more than once. Descriptive statistics were used to explore these routinely collected data. Some comparisons by demographics were carried out by crosstabulation and χ2 tests or t-tests. We used regional population data by age and gender to estimate rates for age/gender groups.15 Two months of patient feedback on three occasions: February–March 2019, 2020 and 2021 (total 2458 records) were thematically analysed by RBJ and JT-R to explore any change in patients’ satisfaction over time. Quantitative satisfaction scores with eConsult (responses on a 5-point Likert scale (very dissatisfied to very satisfied)) were reviewed over time.

Assessment of usability of websites

To better understand how patients could access eConsult we assessed GP websites on three occasions. First, in January 2020, we reviewed 10 GP websites (5/10 highest users and 5/10 lowest users of eConsult in D&C) for how quickly eConsult could be found. Second, we reviewed all 57 Cornish GP practice websites in summer 2021 assessing their conformance to Level AA of Web Content Accessibility Guidelines 2.1 (WCAG 2.1) using the WAVE Evaluation Tool19 automated accessibility checker. We noted the presence or absence of an accessibility widget and the number of clicks to access eConsult and other services on each website.

Ethics

Interviewees gave written consent. Patients opted in at the end of their contact with their practice to give routine feedback to eConsult (online supplemental appendix) and were informed it would be used to improve the system. The website usability studies used publicly available websites and did not require ethical approval.

Patient and public involvement

There was no formally recruited patient or public representative to the team. However, the authors are all users of eConsult and GP websites as patients and other colleagues (users of eConsult and GP websites) have read and commented on the manuscript. Patients were included as participants in the implementation interviews and their eConsult feedback analysed.

Results

Implementation interviews 2018

We focus our results on those aspects related to accessibility, use and usability. Interviews included concerns about workloads that we do not report here.

In June 2018, staff believed eConsult offered patients more choice and convenience. ‘They’ve got something else to offer the patient when there aren’t appointment available, when there is no easy access to the GPs or any health professionals, I think it helps the receptionists to give another thing to the patient’ (GP, surgery 7).

Staff thought eConsult was particularly suitable for younger patients, those that worked or had young children and patients remarked on improved accessibility. ‘You don’t need to sit for ages on a phone waiting’ (Patient, surgery 6) ‘to be able to do it out of hours was good’ (Patient, surgery 1) ‘It saves me driving, that saves a lot of time.’ (Patient, surgery 6); ‘For me it’s enhanced my access to the surgery’ (Patient, surgery 6)

Some patients remarked on the ability to complete eConsult out of hours and at home, the benefits of reduced travel and avoiding the childcare issues of face-to-face appointments. Other patients talked of avoiding having to tell receptionists personal details, not having to take time off work and being able to ask things they would not normally ask.

However, staff had concerns about repeated use and those that might ‘game the system’ by using it as a way to get quicker face-to-face appointments. Patients confirmed that they had altered responses to eConsult questions in order to progress through the system and obtain appointments. For some this meant downplaying their symptoms to avoid a ‘phone 999’ system response. Staff confirmed this, commenting on patients who had telephoned the surgery after trying to complete an eConsult and receiving emergency advice. ‘Certain questions they won’t let you go any further, it says phone 999 or something, and you’re like I’m not dying, so you’ve got to go back and lie to get the eConsult through’ (Patient, surgery 6). ‘They would maybe underplay what they feel, they’re in a lot of pain, but they wouldn’t necessarily say that because they don’t want to call the ambulance, or they don’t want to be directed to 111 or something else.’ (Operations manager, surgery 3)

Some staff described eConsult as being clunky, long-winded and repetitive. Patients agreed. ‘It’s quite long-winded and time-consuming going through all those questions first before you can actually get to say what you want to say’ (Operations manager, surgery 3) ‘I just found it a little bit repetitive, I can understand why the questions are sometimes asked again, but I almost gave up the will to live’ (Patient, surgery 1); ‘the problem is that it gets you to fill in a whole lot of open field, what’s the problem blah, blah, blah, and then it moves into mode where it asks you to choose between a list, and that’s seems to a very large extent repetitive.’ (Patient, surgery 1) ‘I find it slightly cumbersome’ (Patient, surgery 1); ‘I find it clunky to use’ (Patient, surgery 6)

Limited patient involvement in the implementation of eConsult was the most frequently mentioned barrier to successful implementation; interviewees at five of the seven practices reported no patient involvement. Staff suggested that patient involvement would have helped implementation, but it was not possible due to lack of resources. ‘There was no direct consultation with patients before we implemented it. It would have been helpful because there are quite a few of them who are tech-savvy who could say yes I can understand that but actually if you did it this way it would be easier.’ (Practice manager, surgery 2). Other staff did not see the need for patient involvement. ‘We didn’t involve patients no. For us personally it wouldn’t have made any difference because we knew what we wanted.’ (Assistant practice manager, surgery 5).

Where patients had been involved, their experience was positively described. Related to this some practices, in 2018, reported a lack of patient understanding of the purpose of eConsult. Patient expectations about practice response (eg, expecting a quicker response, or being able to do more with the information given) was also an issue. ‘Sometimes patients don’t quite understand what eConsult’s for so they misuse it’ (Admin, surgery 6). ‘I can think of 3 in the last couple of weeks who’ve put an eConsult in and then phoned for an appointment.’ (Admin support, surgery 7).

Patients remarked on particular aspects of the system such as information about when they would be contacted. Patients thought receiving responses within 24 hours helped reduce anxiety and avoided the need to join the telephone queue for appointments. ‘you never when or where or how, so you could be a bit on edge not knowing exactly when they are going to contact you, or who is going to contact you, so you’ve got no idea’ (Patient, surgery 6).

Staff wanted better linkage with existing online GP record systems and for the eConsult and outcome to go automatically into patients’ medical record. ‘I think it could be smarter if it was integrated with system one’ (Admin, surgery 6); ‘It needs to be integrated into System One’ (GP, surgery 6). ‘If it could automatically go into the notes I’d be much more sold’ (GP, surgery 3).

Some patients remarked on the difficulty of finding eConsult online while others remarked that it was prominent. ‘I found it quite difficult to find’ (Patient, surgery 2). ‘I just assumed that if I went into Patient Access I would find an option to go for an eConsult, but I found out later you have to actually go into a website on your own surgery, and that comes up with the eConsult.’ (Patient, surgery 1). ‘it’s quite prominent on the website and when you’re on there sometimes it almost feels like it gets in the way, it pops up when you don’t want it…’ (Patient, surgery 6).

Routine episode (usage) data 2018-2021

There were 922 066 eConsult consultations in D&C between June 2018 and March 2021. Patients’ use of eConsult had been increasing slowly before the pandemic but increased significantly from March 2020 (figure 1). Men’s rate of use of eConsult was under half that of women (47%, 784 vs 1677) per 100 000 in February 2020, and this proportional gap remained as use rapidly increased (46%, 3304 vs 7238 in March 2021). There was no obvious difference in usage rates by age group.

Figure 1

Monthly episodes/100 000 population by gender (blue=women; brown=men) and by age group (blue=0–30 years; brown=31–64; grey=65+) from August 2018 to March 2021.

Routine feedback data 2018–2021

Overall few patients (28134, 3%) provided feedback on their eConsult experience but those who did were mostly satisfied. Mean satisfaction scores did not vary over time, being 4.1 for each year 2018–2021. Most (26232, 93.2%) of the 28 134 responses gave their gender as male or female, 1869 (6.6%) did not answer and 33 (0.1%) were non binary. Of the 26232, satisfaction scores were slightly higher among women than men (4.2 vs 4.0; t=6.3; p<0.001) and slightly higher among the under 65s compared with older patients (4.1 vs 4.0; t=5.2; p<0.001).

Most patients made no suggestions on how to improve eConsult or said that they thought the system was fine (table 1). A small minority were very unhappy with the system. The most frequent comments echoed the implementation interviews concerns on the repetitiveness of the questions. Patients wanted more focused routes through the questionnaire. Others commented that the doctor had not read the eConsult as they asked the same questions again. The issue of having to ‘lie’ to the questionnaire to avoid being told to call 999 was raised by 53/2458 (table 1). Continuity and integration with GP systems was also a problem. Many patients commented on the lack of continuity in the way eConsult operated particularly when they had to complete the form again for a follow-up appointment and had to start the form anew. Others complained about having to give answers about information (such as prescriptions) that should be available in their medical record.

Table 1

Classification of responses to ‘If you could make one improvement to the eConsult service what would it be?’ for 2458 people who completed eConsult feedback form in March 2019, 2020 and 2021

No improvement	1132	No response (box left blank) or said ‘not applicable’, ‘nothing’ or ‘everything fine’
Scrap it	116	People wanted the system scrapped, hated the process.
Practice response (no software change suggested)	153	Comments on the practice’s response to the eConsult, mostly seeking a quicker response, some commenting that the doctor had not read the eConsult (as they asked the same questions all over again).
Practice response (software enhancement suggested)	126	These comments wanted a better response from the practice but suggested ways in which eConsult could enable that for example by specifying a smaller time window for when the GP call would be made, being able to specify that they wanted their own GP to respond, or that the GP could leave a message on an answer machine if the patient did not answer, indicating that a prescription was ready to collect, or that the doctor had read the eConsult but had not yet responded.
Questions and algorithms	415	The most frequent comments were about the questionnaire and the repetitiveness of the questions. Patients wanted more focused routes through the questionnaire. A large sub group (n=53) commented that they need to lie in their answers so as to be able to complete the questionnaire without being told to call 999.
Simpler contact and better integration with GP website	130	Many wanted simpler ways of contacting the practice to request an appointment, a repeat prescription, or medication review suggesting, for example, some way of emailing the practice but as GP websites did not seem to offer this they had to use eConsult. Ten people explicitly asked for eConsult to be easier to find on the GP website.
Design Issues	180	Many had comments on design issues such as the ability to upload pictures or other files. Others had comments on fonts, colours and explanation on eConsult and some (mostly in 2021) wanted the ability to videocall the doctor.
Continuity	106	Many had comments about the lack of continuity in the way eConsult operated particularly when they had to complete the form again for a follow-up appointment and had to start the form anew, repeating information for example about drinking and smoking that they had very recently completed. Others complained about having to give answers about information (such as prescriptions) that should be available in their medical record.
Awareness and help	15	Some expressed the need for better awareness about the use of eConsult and guidance on its use
Accessibility others	23	Some were concerned about the accessibility of eConsult for others who had no computer.
Unclassified	56
Total	2458

Categories highlighted in orange were considered to be possibly addressable by software improvements and were investigated further over these 3 years.

GP, general practitioner.

We reviewed differences over time for those themes where either eConsult might have been able to improve the software, or where the practice may have been able to better integrate its use with the GP website (table 1). There were some but few differences. For example, in 2020 there were eight suggestions about uploading photos and/or a digital voice file. For example ‘Being able to send photos’, ‘To be able to attach files like) a photo or report etc’, ‘It might be useful if you could attach an image or images? Or perhaps a digital voice file.’ In 2021 patients’ comments suggest this may have been added as a feature but that more work was needed: ‘I was asked to send in some pictures, but wasn’t told how to do this’, ‘I had to complete the form twice as when I tried to upload the photos it clicked off the form so had to start again which was time consuming’, ‘I have poor broadband so it was difficult to upload photos’.

Other patients commented on continuity, for which there appeared to be no change. For example, in 2020 comments included: ‘Please don’t make me fill in all the background stuff again - it should just be there because of a cookie’, ‘For a second consultation in the future do you need to know if I smoke etc, You should have this on record’, ‘If the consultants need to see what meds you are on and your family history then they should look up your records, the amount of info I have to give each time is ridiculous’, ‘No need to ask me for a list of my medications—my surgery knows this’. In 2021 however, there seemed little improvement: ‘Less Q&A, as all details needed should be on my records’, ‘Allow a log in account which contains your basic information and reduce the amount of unnecessary questions’, ‘Not have to answer questions about medication I take as the GP has a record of these already’.

Usability of GP websites 2020–2021

Examination of GP websites in January 2020 (pre-pandemic) for five practices with most and five with least use of eConsult demonstrated simple and obvious differences. On the low use websites, eConsult was more difficult to find or had some user barrier to entry. For example, two practices had pop-up windows requiring the user to agree to cookies before they could continue. One practice website required users to first choose online requests from a none-too-obvious menu bar then work out to choose ‘Ask a doctor a question’. Another had the link embedded in a menu of other items, but using a colour scheme that made it ‘disappear’ to the background. By contrast, two of the five high use websites had an eConsult entry page ‘pop-up’ as the main item blocking all other items. Another had eConsult as the main panel central to the opening menu with most other items in smaller font and less attractive as options.

In August 2021, most (52/57) Cornish GP websites were using one of five proprietary systems with five practices developing their websites ‘in-house’ (table 2). Assessment of accessibility against WCAG 2.1 showed that most (38/57) had at least one error and most (39/57) had at least one area of poor contrast. One self-made website had 58 areas of poor contrast and another self-made website had 34 errors.

Table 2

Web Content Accessibility Guidelines (WCAG) 2.1 assessment for 57 Cornish GP practices, showing website supplier, number of WCAG errors, number of instances of poor contrast, number of websites including accessibility widget

Website supplier	N	Errors		Poor contrast		Widget
		Range	Mean	Range	Mean	Present	Missing
Egton	1	(16–16)	16	(0–0)	0	0	1
My surgery	36	(0–24)	3.3	(0–29)	6.8	0	36
Self-made	5	(0–34)	8.2	(0–58)	18.8	0	5
Silicon practice	11	(0–0)	0	(0–7)	6.4	0	11
Surgery web	3	(7–11)	8.3	(2–4)	2.7	3	0
Surgerylink	1	(4–4)	4	(1–1)	1	0	1
Total	57	(0–34)	3.6	(0–58)	6.2	3	54

GP, general practitioner.

While most (45/57, 79%) websites allowed access to eConsult or another triage system in one or two clicks, six took three clicks and six had no link (table 3). Other online services took two or more clicks and only 37/57, 65%) had a link to possible access to patients’ medical records. Most practices used nationally available patient portals (28/57 Patient Access and 19/57 SystmOnline) to provide access to online services. Practices requiring more clicks seemed to have lower rates of use of eConsult (table 4) although this was not quite statistically significant (analysis of variance F=2.6; 3,43 df; p=0.06).

Table 3

Clicks needed to reach online service for 57 Cornish GP practice websites from the GP home page

Online service	No of clicks needed to reach				No link
	1	2	3	4
eConsult or other triage system	15	30	6	0	6
Prescription ordering	2	37	11	0	7
Make appointment	1	36	9	1	10
Access medical record	0	25	12	0	20

GP, general practitioner.

Table 4

Clicks needed to reach eConsult for 47 Cornish GP practices using eConsult in 2021 and for whom website data was available

Clicks to reach eConsult (September 2021)	Mean (SD) no eConsults from 1 January 2021–31 March 2021	No of practices
1	122.8 (115.1)	14
2	58.7 (50.0)	25
3	52.4 (69.2)	5
No link	32.3 (33.7)	3

GP, general practitioner.

Discussion

The COVID-19 pandemic required both patients and GPs to make greater use of online services with mixed reception from both. There is both enthusiasm for the uptake of these new methods and others who have concerns about efficiency (duplication of work and less than satisfactory workflows), effectiveness (how well the clinical encounter works), and in particular, concerns about digital inequalities. Others have discussed the workloads associated with GP triage systems20 and both patient and staff feedback suggest that more can done to improve the use, usability and accessibility of GP online services. We explored four datasets, usage data, implementation interviews, and routine feedback for eConsult and a review of GP websites, to better understand how online services might continue to be improved.

Usage data

Usage data for eConsult confirms what many would claim, that older people have less access online than face to face or perhaps by telephone. We would expect a steady increase in consultation rates by age. Mukhtar et al found (in 2014) that 45–64 years old consulted face to face twice as often, 65–74 year olds nearly three times and those 75 and over almost four times as often as those aged 5–14.21 Yet we found online consultation rates similar for all ages. Most practices retained a variety of forms of access of which eConsult was just one. We do not know what the continued face-to-face or telephone consulting rates were for older patients, but it is clear that online contact was much less than ‘normal’ and so it seems likely that older patients had diminished accessibility online. This corresponds with data for telehealth from the USA.22

A marked finding of the usage data is the much greater use (more than double) of eConsult by women than men. The idea that women consult more than men is indeed not new; in 2010 Wang et al found that men consulted 32% less often than women,23 while in 2018 Mukhtar et al21 found that women’s consultation rates were 21% higher than males. What is perhaps surprising is that this difference is even greater online. Men tend to be earlier adopters of technology24 and some would argue that providing online access to health services might allow more men to consult who would otherwise not.25 Given the big increases in workloads remarked on by GPs,26 this may still be true but it seems that women have taken more advantage of online access than men. There remains a possibility that this is partly artefact with contacts being recorded as women when they were acting on behalf of children. Further research is needed.

Implementation interviews

This is a rapidly changing environment and eConsult (B.Hayhoe personal communication, 2021) argue that our 2018 implementation interviews are now out of date in terms of the product (eConsult), experience of users (patients and clinicians), and the healthcare context (post-NHS England (NHSE) total triage directive). In our interviews, we found that practices had rarely involved patients and eConsult argue that they now provide ‘transformation teams’. However, our interest and focus was on how eHealth systems evolve and what lessons can be learnt by others, so we argue that it was relevant to look at views expressed in 2018 and if/how systems changed in the subsequent years. We did that by examining routine user feedback over 2018–2021.

Routine feedback data

Routine feedback was received on less than 3% of consultations giving a limited understanding of people’s experiences. Systems capturing qualitative feedback, like Care Opinion, can have success in improving services27 but patients need to know that the service is ‘listening’ and responding appropriately to feedback.28 Most of those who provided feedback on eConsult were satisfied, although a minority were very unhappy, about its use. The most frequent comments about eConsult were on the repetitiveness of the questions. Patients wanted more focused routes through the questionnaire. The issue of having to ‘lie’ to the questionnaire to avoid being told to call 999 was raised many times in patient feedback over the 3 years, as it had been in the early implementation interviews. eConsult argue (B. Hayhoe, personal communication) that red flag warnings incorporated into the system are key to patient safety. Further research into why patients have to change their answers to try to avoid these warnings is needed.

Continuity and integration with GP systems also appears problematic. Many patients commented on the lack of continuity in the way eConsult operated particularly when they had to start the form anew for a follow-up appointment. Others complained about having to give answers about information (such as prescriptions) that should be available in their medical record. Patients expect their eHealth interactions to be ‘seamless’ and for systems such as eConsult to communicate effectively with their medical record. This of course is not a problem limited to eConsult or primary care. Both information giving and question asking should be tailored to the patient based on their medical record.29 30

Our examination of patients’ suggestions over time suggests that although some patient feedback is being acted on, much has not. eConsult hold regular meetings at which action on feedback relevant to patient safety is discussed (B. Hayhoe, personal communication) and eConsult provide a log of changes made to the system (https://help.econsult.health/en/articles/5402776-econsult-platform-updates-log) but we are not aware of any current process that lets patients know if their feedback has had any effect regarding continual improvement and evolution of GP online services. (eConsult have provided a list of major changes to the software over the period of study (online supplemental appendix). Care Opinion was established in 2005 and has grown to become a way of gathering and using patient feedback to improve patient-centred services. Previous work has shown the importance of letting patients know that someone is acting on their feedback.28 Currently, the same is not happening for online services. Although eConsult may review sample feedback forms and pass this information to practices, there is no clear indication that it is being acted on; certainly patients are not given responses to their feedback.

Access to eConsult and other services via GP websites

When, in January 2020, we first looked at high and low users of eConsult we found that low users often ‘hid’ the eConsult link in a second or third webpage. Our check on all Cornish websites in summer 2021 seems to confirm that number of clicks to find eConsult may influence its rate of use. Others31 have reviewed the accessibility of GP websites and aim to support accessibility audits, but is it reasonable to expect a single handed or even small practice to afford the cost of a fully accessible and usable website? In November 2021, six website providers supplied most websites but with some still ‘self-made’. The costs to small practices are not insignificant. This could be provided at national or perhaps regional level with ‘drill down’ details for particular practices. Most patients access their GP website not because they want to know the qualifications or recent news of the practice staff but because they want an appointment or advice. Most GP websites now provide such services via national providers.

Research and development

Why women use eConsult far more than men is worth exploring further. More importantly work is needed to create national transparent systems of patient feedback, as Care Opinion does for face-to-face services, for online services such as eConsult. This would allow users to suggest ways of improving systems and know that their suggestions are acted on. Work is needed on eConsult to address issues such as repetitiveness and the need to lie to get the desired outcomes, lack of link between eConsults and the medical record and other patient portal services.

Implications for practice

We question the need for each GP practice to have their own website. Most patients think of the National Health Service (NHS) as one organisation. Anecdotally, we are aware that there is confusion about ‘the way in’ to the NHS and whether to phone the practice, use eConsult, phone 111 or even visit the emergency department. The variable quality of GP websites adds to this confusion. We argue that access to GP services should be via one, well developed and tested, accessible NHS portal or at most a regional of Primary Care Network website. The NHS app is gaining in popularity and use and is likely to become a de facto standard point of entry to online services such as eConsult, patient access to medical records and to good quality websites and apps. Some people may argue that regions such as D&C need different services to an inner urban area but there is likely to be more variation by age and device use than by region. Younger people are more likely to access such services on mobile, perhaps on the move, while older people on tablet or personal computer at home. Developers need to ensure that patients across a wide range of demographics are involved in codesign.

Strengths and limitations

This comprehensive exploration of usability using four datasets from three sources including 3 years of eConsult routine data for D&C has been carried out by independent researchers with no conflicts of interest. It is limited by being a pragmatic approach ‘following the data’ rather than preplanned. The routine eConsult data collected are limited by being based on episodes rather than individuals. In using eConsult feedback data, we overstate the accessibility of online services as only those who have successfully used the service have the option of giving feedback (and of those only 3% gave feedback).

For consistency to mitigate against confounders, our review of GP websites was carried out by one author (IDCM) within a short time frame, using only one device (PC) on a stable connection. This may give an overoptimistic view of the usability of websites. User-experience of GP websites may be worse depending on digital literacy, different devices, and stability of internet connection. A larger study with several raters and with control of these confounding factors may be worthwhile.