Assessing the performances of a chatbot to collect real-life data of patients suffering from primary headache disorders.

Background: There are many scales for screening the impact of a disease. These scales are generally used to diagnose or assess the type and severity of a disease and are carried out by doctors. The chatbot helps patients suffering from primary headache disorders through personalized text messages. It could be used to collect patient-reported outcomes. Objective: The aims of this study were (1) to study whether the collection and analysis of remote scores, without prior medical intervention, are possible by a chatbot, (2) to perform suggested diagnosis and define the type of headaches, and (3) to assess the patient satisfaction and engagement with the chatbot. Method: Voluntary users of the chatbot were recruited online. They had to be over 18 and have a personal history of headaches. A questionnaire was presented (1) by text messages to the participants to evaluate migraines (2) based on the criteria of the International Headache Society. Then, the Likert scale (3) was used to assess overall satisfaction with the use of the chatbot.
Results: We included 610 participants with primary headache disorders. A total of 89.94% (572/610) participants had fully completed the questionnaire (eight items), 4.72% (30/610) had partially completed it, and 5.41% (33) had refused to complete it. Statistical analysis was performed on 86.01% (547/610) of participants. Auto diagnostic showed that 14.26% (78/547) participants had a tension headache, and 85.74% (469/547) had a probable migraine. In this population, 15.78% (74/469) suffered from migraine without probable aura, and 84.22% (395/469) had migraine without aura. The patient's age had a significant incidence regarding the auto diagnosis (P = .008<.05). The evaluation of overall satisfaction shows that a total of 93.9% (599/610) of users were satisfied or very satisfied regarding the timeliness of responses the chatbot provides.
Conclusion: The study confirmed that it was possible to obtain such a collection remotely, and quickly (average time of 3.24 min) with a high success rate (89.67% (547/610) participants who had fully completed the IHS questionnaire). Users were strongly engaged through chatbot: out of the total number of participants, we observed a very low number of uncompleted questionnaires (6.23% (38/610)). Conversational agents can be used to remotely collect data on the nature of the symptoms of patients suffering from primary headache disorders. These results are promising regarding patient engagement and trust in the chatbot.

Introduction

Assessment and follow-up of patients with headache

There are many scales for screening or assessing the impact of disease around the world. These scales are generally used to diagnose or assess the severity of a disease and are carried out by physicians. According to the World Health Organization (WHO), headaches are the most frequent neurological disorder in the world with a prevalence of 50% among adults. This crippling condition ranks third in terms of Disability Adjusted Life Years (DALY), with a cost for society reaching around 173 billion euros per year in Europe. The two main primary headache disorders are migraine and tension headache.

The prevalence of migraine is estimated to be between 17% and 21% in adults aged 18 to 65 years with a sex ratio of three women to one man. According to the diagnostic criteria used: migraines represent 8% to 11%, and probable migraines are 9% to 10%. Tension headache must be distinguished from migraine: a more diffuse headache, non-pulsatile, not aggravated by effort, less intense, and without digestive signs. It can sometimes be accompanied by phonophobia or photophobia, but not at the same time. Despite its high prevalence, migraine remains an under-diagnosed and under-treated condition in the general population, as many patients are often outside the care pathway (extensive use of over-the-counter drugs, no prescription nor recourse to a physician). Migraine can have a significant impact on the patients’ quality of life. Often misunderstood by patients and their families, it can also be confused with other forms of primary headache by patients. The WHO points out the lack of knowledge of health care staff which would limit the correct management of this pathology.

The clinical diagnosis is established by a list of criteria proposed by the IHS (International Headache Society) and published by the French ANAES (Agence Nationale d’accréditation et d’évaluation en santé) in 2002: “Diagnostic and therapeutic management of migraine in adults and children: clinical and economic aspects.” The proper management of migraine is an issue both for patients and for the economy of a country because the economic impact is real and significant due to its high prevalence. Indeed, “migraine is a prevalent and incapacitating condition that affects individuals in the prime of their productive life, thus generating an economic burden for both society and healthcare systems.” Also, patients with migraines used significantly more frequently than controls antidepressants, anxiolytics, and analgesics.

The chatbot as a data collection tool

This situation requires new ways of collecting data and interacting with these patients. Information technology is rapidly expanding and changing the way patients and physicians interact with each other.[9,10] In the field of digital health, a variety of services exist to help patients during their care journey and to connect them with medical staff through smartphone applications.[11-13] Chatbots are an example of these applications. They are software based on artificial intelligence and natural language processing techniques that interact with users via text messages without human intervention. Yet are chatbots good tools to collect and analyses scores? In 2019, Bibault et al. suggested that chatbots could be a solution for monitoring patients during treatment and saving time for healthcare teams for more complex tasks.

Wefight designed a chatbot, Vik, that helps patients suffering from headaches disorders through personalized text messages. Vik provides relevant, quality-checked medical information on this pathology and its epidemiology, treatments, side effects, and strategies to improve quality of life. Chaix et al. showed that it is possible to get support through a chatbot such as Vik by improving treatment adherence with automatic reminders or therapeutic patient education programs. Vik is available for free on the web, on all mobile phones, on iOS (Apple) or Android (Google), or on Messenger (Facebook).

Vik's architecture is composed of several technological parts allowing a fine analysis of the question posed by the user and an adapted treatment for the answer. For a chatbot to be fully developed, both machine learning algorithms and natural language processing are required. To understand the users’ message and send personalized answers, the conversation goes through three steps: the first step analyses the sentence and identifies intents and entities by using machine learning. The second stage activates modules according to the intents and entities detected by the first stage, and the third stage aggregates the answers of all activated modules to build the answer sent to the user and saves the conversation on the user's profile.

For the patient, the use of the chatbot is very simple. He asks a question by writing it on his keyboard, and the chatbot answers directly in simple and understandable language from a database enriched by doctors or pharmacists on various topics mentioned above (Figure 1). The data collected are not commercially exploited, they are anonymized and are used to understand the patient's disease or symptoms in order to refine the accuracy of the answer given by the virtual agent.

Figure 1.

The chatbot's interface.

Objectives

The main objectives of this study (1) were to show that the collection and analysis of remote scores, without prior medical intervention, is possible by a chatbot with a response rate of over 80%, and (2) to perform a suggested diagnosis of the type of primary headache disorder. A questionnaire based on IHS criteria was used. It is simple to handle and allows the type of headache disorder to be defined and rapid diagnosis suggestions to be made.

The secondary objective (3) was to assess the satisfaction of users who have interacted with the chatbot to answer their questions. This study could contribute to the overall knowledge regarding the chatbot in the healthcare field and could be used to assess the performance of this specific tool to collect patient-reported outcomes and analyze remote scores.

Methods

Participants

This study was conducted in France from December 2019 to March 2020. Voluntary users of the chatbot were recruited online through the chatbot interface. They received a message asking them to participate in the study. If they agreed, they had to read and accept the consent form. They had to be adults and have a personal history of headaches. They were not expected to show clinical symptoms outside of the attack phases. Specific inclusion criteria were to have exchanged a minimum of five questions with the chatbot and to have used it for at least 30 days before the study. The exclusion criteria concerned users who were unable to formulate their non-opposition, who had insulted the chatbot, who was under 18 or who had dialogues that made no sense.

Ethical and regulatory issues

The collected data were anonymized and then hosted by Wefight on a server that meets the requirements for storing health data. Consent was collected online before the start of the study. This research was approved by our internal review board. In accordance with French and European laws on information technology and liberties (Commission Nationale Informatique et Libertés, General Regulations for Data Protection), users had the right to access the data to verify its accuracy and, if necessary, to correct, complete, and update it. They also had a right to object to their use and a right to delete such data. The general conditions for the use of the data were presented and explained very clearly. They had to be accepted before accessing the questionnaire. Participants were not paid.

Intervention

A presentation message was sent to the patients, with information about the study and an associated button leading to a questionnaire. It was presented to the participants in order by text messages. Users were asked to click on a button corresponding to the score they wished to give their status (1). There was no actual conversation per question, nor was there a need for natural language processing for each question.

The questionnaire used is composed of eight dichotomous questions, based on IHS recommendations stated by Lanteri-Minet et al. Criteria B, C, and D were therefore adapted to provide information on the correspondence of these benchmarks. The authors are aware that the diagnosis of primary headache disorders relies on a clinical examination performed by a qualified physician; hence the statements further below are suggestions of diagnosis, based on IHS recommendations. The questionnaire was used to define the criteria for migraine. In clinical practice, these questions are asked orally at the time of diagnosis. We used eight questions for which the participant answered by selecting one of the two proposed answers, and each one gets a score associated (Table 1).

Table 1.

Questionnaire used.

Questions	Responses
How does the headache manifest itself?	On both sides (0)	On one side (1)
Does the pain increase when you bend down, when you cough, or when you do physical effort?	No (0)	Yes (1)
How is your pain?	Mild to moderate (0)	Moderate to severe (1)
You feel pain like:	Oppressive (0)	Beating (1)
Do you ever feel nauseated or vomit when you have a headache?	No (0)	Yes (1)
Can you stand bright light when you have a headache?	Yes (0)	No (1)
Are you sensitive to noise when you have headaches?	No (0)	Yes (1)
Does your headache last between 4 h and 3 days without treatment?	No (0)	Yes (2)

The scores were based on the IHS criteria for migraine without aura (Table 2).

Table 2.

Criteria for assessing migraine symptoms.

A At least five episodes meeting criteria B to D.

B Headache episodes lasting from 4 to 72 h (without treatment).

C Headache having at least two of the following characteristics:unilateral;pulsatile;moderate or severe;aggravated by routine physical activities, such as climbing or descending stairs.

D During headaches, at least one of the following characteristics:nausea and/or vomiting;photophobia and phonophobia.

E Clinical examination should be normal between attacks. In case of doubt, the secondary headache should be ruled out by appropriate further investigation.

These criteria did not allow the diagnosis of all migraine cases. We suggest that a score between 0 and 4 would tend to define the attack as a tension headache, while a score between 5 and 9 would be more suggestive of a migraine. In this population of migraine patients, if the patient met criteria B, C, and D, it was a migraine without aura. However, if the person did not meet one of the following criteria: B, C, or D, the migraine was a migraine without a probable aura (meeting all but one of the diagnostic criteria).

During the presentation of the questionnaire, it was specified that the number of episodes presenting these symptoms had to be greater than 5. Through this presentation, we were able to validate remotely the presence of IHS criteria A and E for all participants (2).

The Likert scale ranging from 1 to 5 was then used to assess overall satisfaction with the use of the chatbot (1 “strongly agree” to 5 “strongly disagree”). The timeliness, usefulness, and clarity of Vik's responses were assessed (3) using direct affirmations or questions (“You are satisfied with the timeliness of my answers,” “You’d say I’m like a friend to you,” “You consider me extremely helpful,” “Am I fulfilling your expectations?,” “Am I easy to use?”).

Outcomes

The goal was (1) to measure the percentage of completed questionnaires, the number of people who responded over a given period, and the number of people who agreed to respond in relation to the total number of users over the study period. It was also (2) to perform a suggested diagnosis of the type of the migraines based on the IHS criteria. Then, we assessed (3) user satisfaction (user-friendliness and interactivity).

Statistical analysis

The description of the populations included was carried out by the calculation of mean, standard deviation, median, and quartiles for quantitative variables, numbers, and percentages for qualitative variables.

Classic descriptive analysis and calculations of percentages were used to evaluate the number of completed questionnaires (1) and the overall satisfaction (3). The description of the type of headache was based on the IHS criteria to perform the suggested diagnosis (2). Two groups were defined: “tension headache” and “probable migraine.” The “probable migraine” group was subdivided into two categories: “migraine without probable aura” and “migraine without aura.” The incidence of several variables such as the age, the level of study, and the smartphone/computer usage time over a day, on the division into different groups described above, was assessed using the Fisher test (for the variable level of study) and the Student test (for the two remaining variables). The correlation between these variables and the patient distribution in different groups was declared if the P-value of the test is lower than .05.

Results

Analysis size

We included 610 participants with headache disorders. The median age of the participants was 27 [21-32] years. Females represented 96.89% (591/610) of the participants, 2.79% (17/610) were male, and 0.33%, that is, two subjects answered: “Other.” The educational attainment of the users is shown in Table 3. On average, participants used a smartphone or computer for 5.56 h per day. The average time to complete the satisfaction questionnaire (10 questions) was 3.24 min for all participants (Table 3).

Table 3.

Characteristics of the included participants (n = 610).

Characteristics (n)	% or mean (SD)
Gender
Female (591)	96.89%
Male (17)	2.79%
Other (2)	0.33%
Age (years)	27 [21–32]
Academic level
No diploma (80)	13.11%
First diploma (157)	25.74%
High-school diploma (234)	38.36%
Bachelor's degree (110)	18.03%
Master's degree (25)	4.10%
Doctoral degree (4)	0.66%
Smartphone/computer usage time (h/days)	5,56 (3.65)
Time to complete satisfaction questionnaire (min)	3.24 (1.49)

Principal findings

Data collection of remote scores

A total of 89.67% (547/610) of participants had fully completed the IHS questionnaire (eight items), 4.92% (30/610) had partially completed it and 5.41% (33/610) had refused to complete it. Thus, we performed statistical analysis on 547 participants.

Headaches’ type suggested diagnosis

The 547 participants met criteria A and E according to the inclusion criteria. A total of 14.26% (78/547) scored between 0 and 4 on the IHS criteria-based questionnaire, thus including them in the “tension headache” group. The remaining 85.74% (469/547) were categorized as a patient suffering from probable migraine. Furthermore, the migraine group was subdivided into two groups. A total of 15.78% (74/469) participants met two out of three following criteria: B (having headache episodes lasting from 4 to 72 h (without treatment)); C (having at least two of the following characteristics: unilateral; pulsatile; moderate; or severe; aggravated by routine physical activities)); D (at least one of the following characteristics: nausea and/or vomiting; photophobia; and phonophobia). Therefore, they were labeled as “migraine without probable aura.” The remaining 84.22% (395/469) met all criteria, categorizing them into the “migraine without aura” group.

Comparison of patient groups division

The statistical analysis showed that there is no significant difference in the distribution of the tension headache and migraine groups due to the level of study (P = .286>.05) nor the smartphone/computer usage time over a day (P = .08>.05). However, the student test showed a significant difference concerning the age in the division into these groups (P = .008<.05). We can assume that the age of the patient has an incidence on the score attributed to the auto diagnosis of a patient. Regarding the subdivision of the migraine patient, no significant differences were found due to the age, the level of study, nor the smartphone/computer usage time over a day (P-value were, respectively, P = .203>.05; P = .293>.05; P = .925>.05).

Patients’ satisfaction and engagement with the chatbot

The evaluation of overall satisfaction shows that a total of 94.2% (599/610) of users were satisfied or very satisfied regarding the timeliness of responses that Vik provides (Figure 2). Respondents to “you would say that you consider me a friend” were 33.5% (213/610) to agree or strongly agree and 47.0% (299/610) to disagree or were undecided. They were 45.9% (292/610) to agree or strongly agree and 49.0% (315/610) to disagree or undecided with “you consider me very useful.” Finally, the level of interactivity was good or very good for 69% (439/610) of users who considered that the chatbot met their expectations and 87.6% (557/610) who thought that the chatbot was simple to use.

Figure 2.

Overall satisfaction rating.

Discussion

The aim of the study was to explore whether a chatbot trained to answer questions about headache disorders was able to provide a questionnaire remotely with high patient engagement. We hypothesized that such a virtual assistant built to support migraine patients would be highly interactive, user-friendly, and could retrieve important information remotely such as medical assessment scales. The Vik chatbot we used is available in several versions, covering different pathologies like breast cancer, ovarian cancer, lung cancer, and multiple myeloma. The range of diseases covered also includes chronic affections such as asthma, depression, or migraine. We aimed this study on the Vik Migraine chatbot version for its largest patient cohort, compared to other versions such as Vik Breast or Vik Ovarian, to reach the main objective.

This specific chatbot does not perform diagnosis (section “Intervention”) but is an awareness and follow-up instrumentation about a most disabling and frequent brain disorder all over the world. This application allows users to be informed about the possibility of most common headache disorders and their specificities. Another aspect of this chatbot is its use as a tool to perform follow-up headache characteristics, such as frequency, severity, duration, associated features, and medication responses. It then gives feedback to the users, and thanks to our developers, the data collected could soon be sent to the related physician.

The study confirmed that such a collection was possible quickly with a high success rate (over 80%) and a strong engagement of the users. Indeed, out of the total number of participants, we observed a very low number of uncompleted questionnaires. The overall time to answer the questions was also quite short. The satisfaction rate of the survey was very high for the topics on the quality of the answers provided, the interactivity with the chatbot, and the ease of use. According to Lantéri-Minet, 33% of patients suffering from migraine are not satisfied with the quality of information given by their general practitioner. The feeling that the chatbot is a great help in daily life or that it presents itself as a friend was more moderate. The satisfaction assessment concerns overall satisfaction on specific topics using a Likert scale. While this initial feedback is positive, the authors are aware that further analysis is needed to measure the usefulness of specific clinical information. Moreover, this line of study is being developed in an ongoing study of ours.

We were able to collect useful results from our entire sample. As mentioned before, we selected the chatbot for its largest cohort and are aware that we are not specialists in pathology. Hence, the statistical analysis we performed covered the repartition in a different group and the potential correlation of certain variables to this repartition. The main goal being achieved, we chose to conduct further statistical analysis with the data acquired in more specific studies in the future. By obtaining data from a large sample of users, we will be able to perform qualitative and quantitative analyses to study medication overuse among patients suffering from headaches and find a correlation between several variables and factors involved in headache-related disability.

More than 90% of the chatbot's user base is made up of female users, which has led to a selection bias. Headache disorders mainly impair female patients. This female predominance and the frequent, even elective, occurrence of migraine attacks at the beginning of the menstrual period are generally interpreted as being directly related to hormonal factors. Furthermore, our patient base in the chatbot also has this female preponderance, as it had been shown in the literature that there are differences in technology acceptance between genders. According to Warner et al. “women are more likely to seek health information online than men.” Thus, the chatbot being a type of service to help patients during their care journey and to provide them with health information,[12,13] this female predominance among the chatbot can be explained. The specific inclusion criteria, especially the one implying the use for at least 30 days before the study, led to a second selection bias. Patients were selected on this criterion to make sure that they use the chatbot correctly and then provide qualitative reported outcomes regarding their symptoms. We could suggest that these two selection biases led to different results in remote diagnosis of migraine.

The questionnaire proposed in this study supposed that the patients using the chatbot suffered from either migraine or tension-type headache, while they could suffer from another condition. The critical analysis of the criteria used shows a good specificity but an unsatisfactory sensitivity. These criteria are therefore restrictive and do not allow the diagnosis of all cases of headache disorders. As a matter of fact, other types of primary headache disorders such as trigeminal autonomic cephalgias, epicranial headaches, or other miscellaneous primary headache disorders can occur among our population, but with a smaller prevalence than the tension-type headache or migraines.

The tools we developed provide an indication of the most common type of diagnosis mentioned above, and do not yet allow the characterization of more specific but rare conditions, leading to a potentially small interpretation bias. The Vik Migraine chatbot is updated daily by our healthcare specialists and pharmacists, to enhance the users’ experience with new features, and particularly new information to share with our patients and caregivers. The update of this information is based both on the user's demand throughout the chat and on the recent literature reading. Nevertheless, the main objective of this study was to show whether a remote collection is feasible with a chatbot, and it was reached with an 89.67% success rate.

However, regardless of gender, the benefits of remote data collection are multiple. It allows questionnaires to be sent simultaneously to a large number of patients, the patient's informed consent is guaranteed, and collection times are shorter. Chatbots allow to reach some people who do not attend to medical consultations due to a lack of time or money, as well as a direct interpretation of the results: there is no bias associated with different doctors or technicians collecting or evaluating the data. Finally, these are tasks that can be automated, allowing health professionals to spend more time on other more complex tasks. Another method of automating these various tasks is the use of surveys offered via the Internet or via mobile devices. Kim et al. were particularly interested in the effect of platform and conversational style on the quality of survey responses. They observed a significant difference with high-quality data for the chatbot survey. Several factors come into play regarding this statement, such as the survey's display feature (a conversational interface for the chatbots, whereas the web survey employs a table matrix). The chatbot features create a conversational interactivity, and the survey is no longer perceived as a task but as a social interaction.

The WHO describes the poor management of headache disorders and the lack of information given to patients as well as the lack of training of health professionals on migraine. This chatbot was designed to inform patients. In cancer care, Bibault et al. evaluated the non-inferiority of Vik Sein (another chatbot focused on breast cancer) in providing information versus a college of expert physicians. This chatbot was shown to be capable of providing quality information to patients at least as well as a specialist. This information for patients and the general population is in line with what most countries are doing by emphasizing therapeutic patient education in order to make them actors in their own health. The aim is multiple: to improve patient care and quality of life, and to reduce the costs of treating the disease In 2019, Piau et al. had also shown that a chatbot designed for monitoring older patients with cancer was useful and could be integrated into a complex healthcare system. Their results showed a high level of commitment to this type of solution by reducing the number of support calls to healthcare personnel. New functionalities are planned to confirm Vik as an intermediary between the patient and medical team to provide relevant information to the physicians and enable real-time monitoring.

Conclusion

Conversational agents can be used to remotely collect data on the nature of symptoms of headache disorders patients. We showed that it was possible to obtain such a collection remotely, and quickly (average time of 3.24 min) with a high success rate (89.67% (547/610) participants who had fully completed the IHS questionnaire) and a strong engagement of the users. The evaluation of overall satisfaction shows that a total of 94.2% (599/610) of users were satisfied or very satisfied regarding the timeliness of responses that Vik provides. These results are promising regarding patient engagement and trust in the chatbot, as well as the collection of patient-reported outcomes and analysis of remote scores. However, the limits encountered made us think about biases that can be limited or avoided, notably the selection bias. Further studies will be carried out on data collection in a more representative patient population in another Vik chatbot, to assess the quality and the usefulness of the information and the care given to the patient. In addition, studies will be conducted to validate and evaluate the several functionalities that will be added to Vik, to provide relevant information for both the patient and the physician.