Co-Yudh: A Convolutional Neural Network (CNN)-Inspired Platform for COVID Handling and Awareness.

The COVID-19 pandemic has been a menace to the World. According to WHO, a mortality rate of 1.99% is reported as of 28th November 2021. The need of the hour is to implement certain safety measures that may not eradicate but at least put a restriction on the rising number of COVID-19 cases all over the World. To ensure that the COVID-19 protocols are being abided by, a Convolutional Neural Network (CNN)-based framework "Co-Yudh" is being developed that comprises features like detecting face masks and social distancing, tracking the number of COVID-19 cases, and providing an online medical consultancy. The paper proposes two algorithms based on CNN for implementing the above features such as real-time face mask detection using the Transfer Learning approach in which the MobileNetV2 model is used which is trained on the Simulated Masked Face Dataset (SMFD). Further, the trained model is evaluated on the novel dataset-Mask Evaluation Dataset (MED). Additionally, the YOLOv4 model is used for detecting social distancing. It also uses web scraping for tracking the number of COVID-19 cases which updates on a daily basis. This is an easy-to-use framework that can be installed in various workplaces and can serve all the purposes to keep a check on the COVID-19 protocols in the area. Our preliminary results are quite satisfactory when tested against different environmental variables and show promising avenues for further exploration of the technique. The proposed framework is a more improved version of the existing works done so far.

Introduction

As the numbers of coronavirus cases are increasing at an alarming rate, it has become a tedious task for the desperate government towards bringing out a cure. Till 30th November, 2021 COVID-19 has infected the citizens of more than 212 countries leading to 261,435,768 patients out of which 5,207,634 people had lost their lives, reported by the World Health Organization (WHO) [1].

It is an infectious disease, so to prevent infection and boost the immunity, people are getting vaccinated. As of 28th November 2021, a total of 7,772,799,316 vaccine doses have been administered as per WHO [1]. Simple home remedies such as practicing hygiene, staying indoors, and avoiding crowded places can also help people stay safe. But, at the same time, the World economy has clawed back hundreds of millions of jobs. It is believed that one out of every third person in India is jobless [2] and due to this reason, there is a need to boost the failing economy. It was not possible for every organization in the World to enforce work from home policy. As a result, it becomes a necessity for the employees to work at their respective workplaces to earn a living. It is believed that 65% of the total employees have returned to work at their respective workplaces [3]. Therefore, it is the responsibility of every individual who is working at a workplace to ensure his safety and stop the prevalence of this virus in society. Just by taking certain preventive measures he/she can stop the flow of virus in his/her community.

To slow down the rate of spread of the disease it is necessary to maintain physical distance. It is a must for every individual to maintain a distance of about two meters from every other individual [4]. Hence, maintaining the norm of social distancing became a necessity to live a safer and healthier life. Studies have also shown that the use of face masks reduces the risk of viral transmission [5] as well as provides a sense of protection. However, it becomes impossible to manually enforce such policies at the premises. Therefore, computer vision-based CNN provides a better alternative to this.

Computer vision is used in Human Action Recognition where recognition of each action performed by the human is given a label of people’s actions in the video [6] [7]. Human action recognition is increasingly being used in the field of security where systems check the criminal actions of thieves and terrorists [8]. Computer vision can also be used for Face Mask Detection [9, 10] and Social Distancing Detection [11]. Face Mask Detection refers to detecting whether a person is wearing a mask or not and what is the location of the face mask [12]. The problem is closely related to general object detection to detect the classes of objects and face detection is to detect a particular class of objects, i.e. face [13, 14]. This detector can be easily integrated with image or video capturing devices such as CCTV cameras at the entrances of public places and corporate offices such that when an individual is not wearing a mask, he should not be allowed to enter the premises.

The rest of the paper is organized as follows: the next section deals with motivation and related work. The subsequent section discusses architecture and experimental setup followed by which the features of the application are presented. Experimental results are presented next. Limitations are discussed in the penultimate section. Finally, conclusion and future scope are provided.

Motivation and Related Work

Related Work

In applications of high utility such as video surveillance, face recognition, face image database management, and face recognition, etc, Human Face Detection plays an important role. Deep learning-based methods have shown better performances in terms of accuracy and speed of processing in image recognition as compared to the traditional Machine Learning Approaches [15].

Deep Convolutional Neural Network is the standard approach in the modern era of Deep Learning for image classification problems. Convolutional Neural Network (CNN) is also pertinent for several domains: voice recognition, computer vision [16], audible or visual signal analysis and facial recognition [17], disaster recognition [18]. CNN helps to do business with the challenges of data analysis in high-dimensional spaces by arranging a class of algorithms to unblock the complicated state of affairs and offer noteworthy prospects [19]. CNN structural design largely comprises of three types of layers alongside an input layer which holds the pixel data of the input image [20].

Significant amount of work has been done which involves wide areas of research in the use of new information technologies, particularly the ones where CNN comes into picture. People have investigated the problem of detecting face masks using various deep CNNs to extract in depth features from images of faces. One such work has been done using Support Vector Machine (SVM) and K-Nearest Neighbors (K-NN) [21] and a comparison has been drawn out between the two based on accuracy and performance metrics. Some teams have worked on real-time mask detection [9] by applying the SSDMNV2 approach that makes the use of Single Shot Multibox Detector as a face detector and MobileNetV2 architecture being the framework of the classifier.

Another work known as RetinaFaceMask [22] is a one-stage detector, in which there exists, a feature pyramid network to fuse high-level semantic information with feature maps along with a context attention module that helps in the detection of face masks. Another work in this domain is based on a CNN architecture used for detecting medical face masks [23] for development on resource-constrained endpoints having extremely low memory footprints. Work on IoT enabled smart doors [24] for monitoring body temperature and face mask detection has been done in which the face mask detection is done by a face mask detection algorithm to evaluate the proposed framework. There has been work on face mask detection [10] and movement detection [11] using deep learning in the era of the COVID-19 pandemic. These have been considered as one of the key components in COVID-19 detection and prevention.

Response to COVID-19

China was one of the first nations that took an initiative in response to COVID-19 as it focused on newer Artificial Intelligence applications like facial recognition systems, robots, and drones. The facial recognition systems were used to track the infected patients with traveling history, robots to deliver items that were used for daily needs like food and medicines, and drones to disinfect large areas [25]. Mostafiz et al. [26] used random forest classifier for the detection of COVID-19 in chest detection. They used hybridization of deep CNN and discrete wavelet transform (DWT) optimized features which gave a satisfactory performance with an overall accuracy of more than 98.5%.

Tracking software like monitoring bracelets was developed with the help of AI to help in the classification of people breaking the quarantine rule. In several nations, Smart Phones and AI-enhanced thermal cameras are currently being used to detect fever and infected people in many countries across the World [27]. Nations like Taiwan resorted to other techniques as their administration maintained a national medical insurance database with data from the immigration and customs. This was further used to detect the people having COVID-19 symptoms through their traveling history [28]. Loey et al. [10] proposed a hybrid deep learning model with machine learning methods, trained and evaluated on three face mask datasets, and showed promising results. Another prominent study of CNN-based mask detection was proposed by Suresh et al. [29]. They implemented optimized CNN on datasets acquired from Kaggle.

The E-Commerce giant JD.com’s [30] efforts are unmatched in delivering essential goods across major cities in China to fight the COVID-19 pandemic. The local government has also played a supporting role by giving an allowance to the company in deploying drones to conduct surveys, designing flight corridors, and conducting flight tests in the country. In Inner Mongolia, JD.com has done a commendable job in bringing laborers back to work by deploying a bunch of drones to support critical disinfection techniques by spraying premises in the High-tech Industrial Development Zone of Ordos City. The World Health Organization (WHO) and other global health organizations are working hand in hand as the need for developments in the healthcare industry is a priority [31].

The concept of Fangcang shelter hospitals first implemented in China in February, 2020 has been adopted by many nations to tackle the pandemic. In this concept, open space public places such as stadiums, exhibition centers are converted into health-care centers [32].

Experimental Setup

Architecture

The architecture consists of four parts as shown in Fig. 1. The landing page displays the cases’ information of COVID-19. On the same page, there are three buttons on the navigation bar that lead to Social Distancing Detection, Face Mask Detection, and online email service page. All these components are explained in detail in further sections.

Fig. 1

Architecture of application

Transfer Learning from MobileNetV2

Deep neural networks are used for image classification because of their better performance than other algorithms. But training a deep neural network is expensive because it requires high computational power and other resources, and it is time-consuming. To make the network to train faster and cost effective, deep learning-based transfer learning is evolved. Transfer learning allows to transfer the trained knowledge of the neural network in terms of parametric weights to the new model [33].

In this application, based on the transfer learning approach, utilization of MobileNetV2 pre-trained model is used to detect people wearing a mask. MobileNetV2 builds upon the ideas from MobileNetV1, using depth wise separable convolution as efficient building blocks [34]. The architecture of MobileNetV2 is explained in [35]. This model is further fine-tuned by adding 7 more layers. The layers added are average pooling layer with a pool size equal to , a flattening layer, followed by two dense layers of 128 neurons with ReLU activation function and dropout rate of 0.5, and finally the decisive dense layer with two neurons and softmax activation function is added to classify whether a person is wearing mask. The model is trained for 25 epochs, each epoch having 34 steps. The schematic representation of the proposed methodology is shown in Fig. 2.

Fig. 2

Face mask model representation [13]

Face Mask Detection

For Face Mask Detection, we have used “Convolutional Architecture for Fast Feature Embedding (CAFFE)” [36] which is a pre-trained model in OpenCV [37] to identify faces. It is likely the fastest available implementation of these algorithms, making it immediately useful for industrial deployment [36].

In this model implementation, a Simulated Masked Face Dataset (SMFD) is used that consists of 1570 images that consist of 785 simulated masked facial images and 785 unmasked facial images. As it can be seen from the dataset description that the amount of training data is limited due to the privacy and security norms, thus it is difficult for our Deep Learning Model to train. Therefore, we used the concept of transfer learning of MobileNetV2.

CNN-based algorithm of Face Mask Detection module is shown in Algorithm 1 and the corresponding process overview is shown in Fig. 3.

Fig. 3

Process overview of face mask detection

Social Distancing Detection

To implement Social Distance Detection, we have used the YOLOv4 [38] model for detecting people as it produces less false positives in comparison to other object detection algorithms. There are various other benefits of YOLO over other object detection models which are discussed in [39]. We also tried various other models like OpenCV’s haar-cascade for detecting pedestrians [40] but it gave more false positives, and YOLOv3 [41] which gave good results but was slower than YOLOv4. Comparison between different models for pedestrian detection is shown in Fig. 4 and test results on sample video is shown in Table 1. Models compared include YOLOv3, opencv’s haar-cascade and YOLOv4. Parameters on which they are compared are accuracy (no. of people detected/ no. of people in the frame), speed (no. of frames processed per second) and no. of false positive (detecting a person which is not actually a person). The main advantage of YOLO is that it is fast and produces less false positives; therefore, it can be applied on a live video.

Fig. 4

Comparison between different models for pedestrian detection

Table 1

Test results for different models on a sample video

Models used	People in frame	People detected	FPS	False positives
YOLOv3	7	7	12	None
Haar-cascade	7	3–4	20	1–2
YOLOv4	7	7	13	None

CNN-based algorithm of Social Distancing Detection module is shown in Algorithm 2 and the corresponding process overview is shown in Fig. 5.

Fig. 5

Process overview of social distancing detection

Scraping COVID-19 Statistics

Extracting useful information from the web is the most significant issue of concern for the realization of semantic web. This may be achieved by web scraping as shown in Fig. 6. Web scraping [42] is a technique of automatic web data extraction to extract data from the HTML of a website by parsing the webpage [43].

Fig. 6

Web scraping [45, 46]

In our application, the information about the coronavirus cases, i.e. active, discharged, death cases of India are scraped from the ministry of health and family welfare’s (mohfw), the official health website of India [44].

Features

Mask detection: It helps to detect face masks. If no mask is found, then the application shows the red warning. If the mask is detected, then the application shows the green signal and allows the person to enter the premises.

Social distance detection: The aim of Social Distance Detector is to raise an alarm/warning when it detects that social distancing protocols are being violated. After locating all the people in the frame, the distance between every pair of persons is calculated (here distance is the Euclidean distance between two points) and if that distance is less than a set threshold value, it raises a warning sign. It also highlights the people violating the social distancing protocol with a red bounding box.

COVID cases tracker: This app provides the information about the coronavirus cases, i.e. active, discharged, death cases of India by web scraping the ministry of health and family welfare’s (mohfw), the official health website of India [44]. It also provides a facility to view date and time. It allows the user to refresh the data. The app uses a beautiful soup library in python language to web scrape the data. The coding has been shared as shown in Fig.7.

Fig. 7

Web scraping using ’beautifulsoup’ library

Online medical consultancy: The Email service offered by this web application allows the members of the organization to contact a government doctor online if he/she is feeling sick or showing COVID-19 symptoms. Mailer is built using the ’smtplib’ library. The coding has been shared as shown in Fig. 8.

Fig. 8

Code of mailer using ‘smtplib’ library

Results

Face Mask Detection is evaluated on a novel dataset - Mask Evaluation Dataset (MED). This data set is constructed by the authors and consists of 57 videos with varying and challenging six parameters. The parameters and their values for evaluations are given in Table 2. Lightning conditions and background were classified into two major categories—shady, good and textured, plain, respectively. Since, beard and spectacles have a major impact on a person’s appearance thus, these both along with gender were also considered. There were different types of masks available, we chose the four most common types of masks used by the people for this dataset. The results of these attributes were categorized into 5 different categories—poor, below average, average, good and very good. The accuracy between 95% and 100% was considered as very good followed by 90% and 95% as good, 85% and 90% as average, 80% and 85% as below average, and below 80% as poor.

Table 2

Attributes and their values for face mask detection test

Lighting condition	Background	Gender	Spectacles	Type of mask	Beard
Shady	Textured	Male	Yes	Surgical mask	Yes
				Cloth mask
Good	Plain	Female	No	Handkerchief	No
				N-95 mask

The graphical analysis of the results considering the individual attributes is shown in Fig. 9. Figure 9a shows different results that were observed on varying lighting conditions. It was observed that the results were average in shady conditions whereas they were very good in good lighting conditions. The model worked very well on both types of background as well as gender as shown in Fig. 9b, c, respectively. There was a significant difference in the results on the spectacles parameter as shown in Fig 9d. Detection of faces without spectacles was very good as compared to its counterpart. Figure 9e shows that the model was able to decently detect the faces in different types of masks with the handkerchief being the most optimally detected type as compared to other masks. Another parameter was beard, where different sets of observations were made on people with and without beard as shown in Fig. 9f. The results of people having beard was little vacillating, whereas it was very good for beardless humans. Therefore, considering all the attributes together, the results were very good as shown in Fig. 10. While checking the accuracy, very good, good, average were considered as positive cases of detection and thus Face Mask Detector gave an accuracy of 91.2%.

Fig. 9

Graphical analysis of discrete attributes

Fig. 10

Graphical analysis considering all attributes

Few results of the face mask detection are as shown in Fig. 11.

Fig. 11

Few results of the face mask detection

We have tested social distancing detection on different types of videos, such as CCTV footage and from cell phones; videos taken from some height and videos taken from ground; videos in natural and artificial light; and also in different crowd levels. The detector fared really well almost in all the conditions but we could see an obvious difference in accuracy between the videos taken from some height and the ones taken from ground level. Videos taken from height (approx. 15 feet) gave better results in comparison to the videos taken from the ground in all conditions.

Social Distancing Detector first locates all the people in the frame and after locating all the people, the distance between every two individuals is calculated (here distance is the Euclidean distance between two points with single camera depth calibration) and if that distance is less than a set threshold value, it raises a warning sign. It highlights the people violating the social distancing protocol with a red bounding box.

Few results of the Social Distancing Detection are as shown in Fig. 12.

Fig. 12

Results for social distancing detection

Limitations

The Face Mask Detector does not give proper accuracy for a person having both spectacles and beard. It also does not show good results when the light source is just behind the person.

When the camera angle is at a perfect side view in which the camera is near to the ground, we get the wrong distance estimation using Euclidean distance and hence Social Distancing Detector may give some false results.

Due to heavy processing during the execution of the social distancing module, the computation requirement of the computing systems is quite high.

Conclusion and Future Scope

This is an autonomous, multi-purpose application that can be used to keep a check on various protocols being followed. The application also provides the facility of tracking COVID-19 (death, active and recovered) cases and an online medical consultancy service. Currently, we are working on the hand hygiene detection application that can be integrated with our system at the entry points of an organization. In the future, we will be working on the mobile application of this framework where the members of an organization will be sent an immediate alert on their mobiles for not wearing face masks. That means the Face Mask Detection application will not only be installed in CCTV cameras at entry points but also on the premises too.