SMAD: SMart assistant during and after a medical emergency case based on deep learning sentiment analysis: The pandemic COVID-19 case.

The huge cost of emergency situations could have fatal effects on humanity and society, and it could present a genuine threat to both of them. In fact, most people confronted with an emergency could feel psychological trauma, which will, for the most part, change over time as they can exhibit chaotic or even turbulent behaviours. The situation could worsen in the case of a pandemic as fear and anxiety invade and spread in addition to isolation and quarantine. In this paper, we propose to build a smart assistant, called SMAD, that could detect the symptoms of an emergency case as well as symptoms of a mental disorder while analysing the natural language speech of an ordinary citizen, during and after an emergency situation using natural language processing and deep learning sentiment analysis model to track the patient's mental state during an ongoing conversation. Our proposed smart assistant is an online human-bot interaction that could handle a variety of physical and mental circumstances of any emergency situation. The proposed approach is a smart healthcare service that consists of four interconnected modules: The information understanding module, the data collector module, the action generator module, and the mental analysis module, which is based on the sentiment analysis model performed on a social media dataset using a pre-trained word-embedding model.

Introduction

According to the World Health Organization (WHO), one in eleven people who have lived through a war or other form of conflict or emergency in the previous ten years will suffer from a moderate or serious mental disorder.1 In fact, most people could experience shock and/or denial during and after experiencing an emergency. In such cases, they show sadness, frustration, anxiety, anger, etc for several hours and sometimes days.2 If these emotions continue, they may lead to more severe mental health issues such as post-traumatic stress disorder (PTSD) and depression. Consequently, providing assistance in an emergency is difficult, and providing it while considering the mental aspect is even more difficult.

Given the aforementioned circumstances, this paper proposes a smart assistant, called SMAD, to help ordinary people overcome an emergency situation both physically and mentally. This assistance will take the form of a natural language conversation. Our assistant’s smartness aspect will allow it to endure several challenges, including understanding the user’s intent through natural language processing, recognizing the emergency, responding with the appropriate medical gestures, and tracking the patient’s mental state during an ongoing conversation through a deep learning sentiment analysis model based on a pre-trained word-embedding.

The proposed approach is a smart healthcare service that is made up of four interconnected modules: the information understanding module (IUM), in which the natural language processing (NLP) is performed to help the assistant better understand the user’s input (voice/text), the data collector module (DCM) that gathers user’s non-confidential information to be used later by the action generator module (AGM) to generate and manage the assistant’s answers. And, finally, the mental analysis module (MAM) that detects alarming behaviours in the text or voice input through a deep learning sentiment analysis model to help AGM deliver more adequate responses such as asking the patient a predefined set of questions from one of the validated scales for depression. If the patient’s depression score, according to the used scale, seems to be very high, the patient should be referred to a list of the closest psychiatrists to his/her location. In the meantime, to alleviate stress, anxiety or depression, our assistant could provide some insights and positive responses.

The remainder of the paper is organized as follows: First, we enumerate some solutions based on NLP and sentiment analysis model proposed to fight emergencies, in 2. Next, we explain our approach through a global architecture and the articulation between the different modules of our assistant 3. Then, we evaluate our work through a test case: COVID-19 in 4. Finally, we end the paper with a conclusion and an outlook on future works for interesting research directions 5.

Related works

Virtual assistants prove their potential in the healthcare field. In our research, we found some papers that are similar to our proposal. Such as SPeCECA [1], which is a smart pervasive chatbot for emergency case assistance and designated to interact with ordinary citizens to help them overcome emergency situations by suggesting the accurate first aid actions to be taken. Additionally, Mandy [2] is a chatbot that communicates with normal people using natural language in order to provide online healthcare suggestions. In [3], a chatbot is designed to mimic human interaction in a medical case. It aims to help ordinary people choose the most appropriate way to prevent disease. Finally in [4], a mobile chatbot is developed for health care service. This chatbot is able to collect and detect the user’s daily health data, in order to diagnose chronic diseases, and provide preventive information and fast treatment after accidents that may occur in everyday life. A Japanese based Bespoke company used artificial intelligence to launch an online chatbot called Bebot that provides up-to-date information on the COVID-19 and preventative actions with a possibility to check symptom [5]. However, all of the papers cited above neglect the psychological aspect during an emergency and/or while providing health assistance.

The second part of our research studies the impact of virtual assistants and NLP to overcome stressful situations. In [6], the authors built a sentiment analysis model based on the bidirectional encoder representations from transformers (BERT) [7] to study the impact of the COVID-19 in social life. The main purpose of this model is to gain insight into people’s mental states via their publicly-posted opinions on a social media platform. For instance, a messenger chatbot was built to recognize and manage emotional stress through a predefined set of questions [8]. Based on almost the same principle, [9] presents a chatbot for mental health counselling also using a set of questions to assess a user’s depression without a deep learning sentiment analysis. However, other works [10-14] have benefited from the power of deep learning sometimes to adjust the sentiment of the chatbot response, and sometimes to detect depression in text sequences. The evaluation of most of these models, which are based on LSTM [15], was made using the performance Measures (recall, precision, accuracy and F1-score) as shown in Table 1.

Table 1

Performance measures of sentiment analysis models used in the literature

Reference	Precision (%)	Recall (%)	Accuracy (%)	F1-score (%)
Singh et al. [6]	–	–	92	–
Chih-Wei et al. [11]	–	–	87	–
Hanai et al. [10]	71	83	–	77
Yenter et al. [12]	–	–	84.98	–
Wang et al. [13]	–	–	90.9	–

Table 2 shows a comparative analysis of some research works based on NLP according to several criteria: virtual assistant-based (VA), emergency case assistance, physical assistance-based, psychological assistance-based

Table 2

Summary of related works

Reference	Virtual assistant	Emergency case assistance	Physical assistance	Psychological assistance		Beneficiary
				QA	SAM	Expert	Ordinary Citizens
Singh et al. [6]		✓			✓	✓
Bespoke [5]	✓	✓	✓				✓
Ouerhani et al. [1]	✓	✓	✓				✓
Chih-Wei et al. [11]	✓				✓	✓
Hanai et al. [10]					✓	✓
Chung et al. [4]	✓		✓				✓
Park et al. [8]	✓			✓			✓
Amato et al. [3]	✓		✓				✓
Lin NI et al. [2]	✓		✓				✓
Cameron et al. [9]	✓			✓			✓
Yenter et al. [12]	✓				✓	✓
Wang et al. [14]	✓				✓	✓
Wang et al. [13]	✓				✓	✓

(Question/answer(QA)/sentiment analysis model(SAM)), Beneficiary(expert/normal citizen).

From the research we have conducted, it is becoming more common to use the power of NLP and virtual assistants to provide physical healthcare assistance [2-4], even for emergency cases [1, 5]. On the other hand, NLP is also used for a psychological assistance through the use of text sentiment analysis systems [6, 10, 11], others through a predefined set of questions [8, 9].

Another relevant fact is the emphasis on providing services to ordinary citizens [1–5, 8, 9] while minimizing direct contact with health professionals [6, 10, 11]

However, the use of NLP for physical and psychological assistance for emergency cases is almost non-existent.

For this reason, we seek a smart assistant to ensure communication and interaction with ordinary people using natural language processing (NLP) methods and based on a deep learning sentiment analysis model in order to assist them physically and psychologically during and after experiencing an emergency situation.

During an emergency situation, an overgrowth of fear and anxiety is noticed whether for the patient or his companions. So, it is necessary to maintain calmness in order to aid the patient effectively and successfully. After experiencing an emergency, it is often critical to provide psychological assistance to the patient in order to avoid aggravating the situation. Therefore, our main contribution focuses on the use of NLP for the physical and psychological assistance of a patient in an emergency situation. In fact, our assistant could recommend the accurate first aid measures to take in any given emergency situation. Furthermore, a sentiment detection system is being developed to track the patient’s psychological state. If our system predicts the possibility of depression, our assistant’s responses change and a depression screening scale is launched to calculate the depression score. According to the used scale, in case of a serious depression, a professional psychotherapy intervention is required. As a result, our assistant advise the patient to visit the nearest psychiatrist promptly depending on his/her location. This process can be tailored to the specific emergency situation.

SMAD architecture

Our adopted approach is organised in a modular manner. To improve the consistency of the implementation, the individual modules of the workflow have been separated into self-contained phases. SMAD’s architecture is shown in Fig. 1, so we divide it into four main modules: information understanding module (IUM), action generator module (AGM), data collector module (DCM) and mental analysis model (MAM).

Fig. 1

Architecture of SMAD

Information understanding module (IUM) SMAD receives the input message sent from the user through IUM. The input could be either a voice or a text. The first task IUM have to do is the input recognition. As a result, there are two processes:Then IUM has to transform unstructured text to a structured representation composed of entities and intents through the Text Analysis sub-module. For this reason, the NLP is used. We have grouped the various steps of NLP into three categories as shown in Fig. 2:

Fig. 2

UIM architecture

Text input process IUM transmits the input straight to the text analysis.

Voice input process IUM must convert speech to text before passing it on to the text analysis.

Lexical analysis Includes tokenization [16] splitting the text into tokens (punctuation marks, numbers, words, etc).

Parser includes

Part of speech tagging (PoS tagging) [17] The process of assigning parts of speech (noun, proper noun, verb, adjective, determiner, and so on) to each token.

Stop word filtering These stop words do not add anything to the knowledge base and can be removed from the text. However, there is no universal list of stop words. The list of words to avoid varies depending on the current field of work.

Lemmatization and stemming [18]

Entities extraction [19] Aims to limit the task to entities that could be important for the next step.

Semantic analysis Recognition of the goal that the user is attempting to achieve. This problem is treated as a multi-classification one, with classes representing the set of possible user intents.

Mental analysis module (MAM) We created a sentiment analysis model to solve the sentiment detection problem. Sentiment analysis [20] is the process of computationally recognizing and classifying sentiments expressed in a piece of text. The sentiment analysis is a multi-class classifier:

Positive When the person presents the following emotions: happiness, joy and love as shown in Fig. 3

Fig. 3

Positive emotions

Negative When the person presents the following emotions: sadness, anger and fear as shown in Fig. 4

Fig. 4

Negative emotions

Neutral When the person presents no specific emotions as shown in Fig. 5

Fig. 5

Neutral emotions

We want to detect and predict the user’s emotions while communicating with our assistant. Rather than learning the response of immediate dependency, our model learns the response of long-term dependency. The MAM’s output is stored in DCM as the user’s mental status which is later used by the AGM. As regular recurrent neural networks (RNN) [21] are difficult to train due to exponential growth from repeatedly multiplying gradients over long sequences [22], we used the long short term memory (LSTM) [15] network. We chose the LSTM because it outperformed gated recurrent units (GRU) [23] in our tests. Thus, as shown in Fig. 6, we developed a sequential neural network model composed of four layers: one embedding layer and one LSTM layer followed by two fully-connected layers.

Fig. 6

MAM architecture

We used the sigmoid activation function [24] as it can turn the output values into a value between 0 and 1.

Action generator module (AGM) SMAD has to provide precise, effective, and rapid actions after recognizing and understanding the user’s request. The action generator module (AGM) is in charge of this task. Since we have found no publicly-accessible conversational data sources between doctors and ordinary people about emergencies, AGM is trained on customised data that we have created from scratch. As this is a classification problem, we use the decision tree algorithm [25] to generate actions. The actions delivered by SMAD are dependent on the user’s input. We have divided AGM into three sub-modules:To speed up interactions between SMAD and the user, there are answers suggested to the user to control the conversation in order to minimize errors as much as possible.

Emergency recognizer: This sub-module is trained to recognize the emergency using symptoms stored in the DCM.

Daily medical checker: After an emergency, a medical check-up is unquestionably the best way to get a full picture of the patient’s health and allow early detection of complications. The data will then be stored in the data collector module, so that experts (doctors, data scientists, and so on) will be able to access it.

Depression detector: AGM will be able to detect the user’s actions during a conversation due to the MAM output. If the user sends three successive negative messages, AGM will try to make the user feel better by giving him or her reassuring messages or pleasantries, even asking him or her why he or she is in such a bad mood. However, if the bad behaviour persists throughout the conversation, AGM will immediately start one of the validated depression scales such as the depression anxiety stress scales-21 (DASS-21) [26], Montgomery–Asberg depression rating scale (MADRS) [27, 28], Hospital anxiety and depression scale (HADS) [29], Beck depression inventory (BDI) [30], patient health questionnaire (PHQ-9) [31], etc. The use of a validated scale ensures the assessment of depression, anxiety, stress, and other related scores in order to assess the intensity of the user’s status in those parameters. This will allow the AGM to determine whether or not the user is depressed.

Data collector module (DCM) The purpose of DCM is to collect important non-confidential data (mainly symptoms) from users and create a dataset that can later be used in machine learning. This information could have an impact on SMAD’s response.

Validation and implementation: the COVID-19 case

COVID-19 is the abbreviated name of the current 2019 coronavirus outbreak, which is a respiratory disease that spreads from person to person.3 This outbreak is the seventh member of the coronavirus family that infects humans, unlike both MERS-CoV and SARS-CoV [32, 33]. It was first reported in December 2019 in Wuhan City in China.

Up to February 2022, almost 5 Million people died worldwide because of the COVID-19, with more than 390 Millions infections reported in most countries, according to the World Health Organization. As a result, the COVID-19 has been declared a pandemic.4

The COVID-19 pandemic has had a significant impact on global mental health, including anxiety, depression, trauma, and stress. It has also led to people fearing and panicking about COVID-19 infection [34, 35]. Worse still, suicide can be triggered by such mental health issues [36].

The following are the key goals of SMAD in case of a pandemic:Our solution is cloud based. Therefore, we developed three separate, but interdependent applications: The mobile and web applications used by citizens and the cloud-based platform, where each of the proposed modules is deployed as a web service. Our web services are exposed as REST API micro-services on the cloud, referenced via their URI as shown in Fig. 7.

Fig. 7

SMAD’s data flow architecture

To raise awareness, but still send out reassuring messages to encourage people to take the necessary precautions against the pandemic.

To collect data from users (non-confidential information) for later use in machine learning.

To instruct both uninfected and infected individuals on how to defend themselves and their companions against further infection.

To determine whether or not the person is depressed and help him/her cope with stress in such a situation, otherwise, if a serious depression is detected, recommend that he or she have to consult a psychiatrist without delay.

The IUM, as explained in Sect. 3, focuses on teaching the assistant how to interpret and understand user’s inputs. We build a natural language understanding (NLU) model and feed it with training data to accomplish this task. After that, the model can learn to convert the data into a structured format with intents and entities. The user messages would then be classified into one or more user intents. We have already chosen to work with the spacy_sklearn pipeline, so that this intent classifier loads a pre-trained language model, which is then used to represent each word in the user’s message as a word-embedding vector. Word-embedding models have been trained on massive text corpora created from Google news and similar sources so the representations may not always transfer well to specific domains. For example, the word “python” means a very large snake in the everyday context while it means a programming language in the field of computer science. These differences become even more relevant in our case because we are supposed to analyse medical data. Our solution is to simply train the global vectors for word representation (GloVe) [37] on our domain-specific data. Since the embeddings are already trained, the SVM [38] requires only little training to make confident intent predictions.

So far, the assistant has been able to understand and process the user’s input. The next step is to train the AGM to teach SMAD to respond to messages. Scenarios are the training data for AGM. The user’s inputs are represented by intents and their corresponding entities, while the assistant’s responses are represented by actions. Finally, we will train AGM based on the policies that should be followed. Because there are no publicly accessible conversational data sources between doctors and ordinary people about COVID-19, AGM is trained on customised data that we created from scratch. As this is a classification problem, we use the decision tree algorithm [25] to generate actions. Since the actions delivered by SMAD are dependent on the user’s input, we have divided AGM into three sub-modules:When the conversation starts, DCM gathers entities and stores them to create a dataset:As for MAM, we used the deep learning library Keras [39]. This is where we get to use the LSTM layer. As we said in 3, our model is composed of an embedding layer, LSTM layer and two dense layers, which form a fully-connected neural network. The first dense layer is based on a ReLU activation function aiming to reach higher accuracy. This layer has no parameters. The second is a sigmoid dense layer. We added dropouts between layers to avoid over-fitting. We chose a dataset from the social media world due to the fact that people nowadays express themselves through these platforms, especially Twitter. On a public Twitter dataset that includes 14,375 data with labelled sentiment (neutral, negative, positive), the sentiment analysis model is performed using the GloVe [37] model.

Daily medical checker: The user who can use this feature must be infected or is suspected of being infected with COVID-19. During the 14-day quarantine period, the patient has to complete a virtual form, which differs from a traditional form in that there are no boxes to fill in, instead the patient has to answer a series of questions in a well-determined order. To put it differently, it meant to keep track of how the different symptoms are progressing.

Depression detector: The chosen evaluated scale is DASS-21 as it allows the measurement of depression, anxiety and stress scores.

Location

Symptoms (fever, cough, dyspnea, ageusia, anosmia, etc)

Age

Gender

Status (infected, not infected/suspected to be infected)

Contact with infected person

Recent travel

Chronic diseases (Alzheimer disease and dementia, arthritis, asthma, heart disease, cancer, diabetes, etc)

User’s mental status

After training and testing our model we got 92% of accuracy and 80.78% of F1-score on a validation set. When we measured the number of correct predictions, we found that negative messages work better for the Network than deciding the positive ones as shown in Table 3.

Table 3

Accuracy of positive and negative messages

Positive messages accuracy (%)	Negative messages accuracy (%)
73.23	97.27

We evaluated our model’s results on the basis of F1-score and accuracy. F1-score, as the primary performance measure and accuracy as the secondary one. To calculate the F1-score we had to calculate first precision and recall that are based on true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN).

Here are the equations that we used:

As shown in Table 4, when we use LSTM Layer we get an F1-score of 80.78% whereas GRU has achieved a lower F1-score of 77.75%. The accuracy is 92% for both LSTM and GRU models. For this reason, we opt for the LSTM layer for our model.

Table 4

Results

Performance metric	Precision (%)	Recall (%)	Accuracy (%)	F1-score (%)
Model based LSTM	86.62	75.70	92	80.78
Model based GRU	86.74	70.46	90	77.75

Also, we can observe low recall, high precision: This shows that we missed some Correct answers.

When compared to other LSTM-based models mentioned in Sect. 2, our model is among the best in terms of accuracy.

To sum up, SMAD keeps requesting information from the user to understand his/her situation and to collect information that will be used for further machine learning. Once the determination is complete, the actions suggested by SMAD are performed. Figs. 8, 9 and 10 demonstrate an example of an executed scenario:

Fig. 8

The mobile SMAD’s responses examples

Fig. 9

The web SMAD’s responses examples

Fig. 10

The web SMAD’s responses: mental health check

Conclusions and future works

In this paper, we presented SMAD: smart assistant during and after a medical emergency case based on a deep learning sentiment analysis. The main purpose of SMAD is to assist a normal citizen to overcome an emergency situation physically and mentally. We build a deep learning sentiment analysis model to track the user’s psychological state. When bad behaviour occurs and persists in an ongoing conversation, SMAD will execute a questionnaire based on validated scales to determine if the bad behaviour is related to a mental disorder or not. SMAD is presented through its four modules: information understanding module (IUM), action generator module (AGM), data collector module (DCM) and mental analysis module (MAM)in which a sentiment analysis model based on the LSTM neural network was applied.