Framework for Real-Time Detection and Identification of possible patients of COVID-19 at public places.

The novel Corona Virus (COVID-19) has become the reason for the world to declare it as a global pandemic, which has already taken many lives from all around the world. This pandemic has become a disaster since the spreading rate from person to person is incredibly high and many techniques have come forth to aid in stopping the infection. Although various types of methods have been put into implementation, the search and suggestions of new approaches to reduce the increasing rate of infection will never come to an end until a vaccine terminates this pandemic. This study focuses on proposing a new framework that is based on Deep Learning algorithms for recognizing the COVID-19 cases, mostly in public places. The algorithms include Background Subtraction for extracting the foreground of thermal images from thermal videos generated by Thermal Cameras through the Thermal Imaging process and the Convolutional Neural Network for detecting people infected with the virus. This automated prototype works in a real-time scenario that helps identify people with the disease and will try to trace it while separating them from having any other contact. This proposal intends to achieve a satisfying growth in determining the real cases of COVID-19 and minimize the spreading rate of this virus to the max, ultimately avoiding more deaths.

Introduction

The whole world has been in chaos due to the outbreak of the COVID-19 pandemic. The Corona Virus disease had been first reported in Wuhan province, China in December 2019 and rapidly spread worldwide within a span of approximately four to five months of its beginning. Since then many precautions and methods have emerged to prevent the spread of the disease [1]. But it seems that the disease is inevitable once it gets into contact with anyone. This has become a major issue as the rate at which the virus is escalating is beyond control and the fact that most of the infected people are not immune to it makes COVID-19 the most dangerous virus to date. The virus itself is so contagious that it may remain airborne for a long period, infecting as many people as it encounters. The symptoms of this virus take several days to first appear and people may also be asymptotic, passing the virus on without knowing that they are carrying it.

To slow down the speed of infection, detection and diagnosis of the diseased people is the first and main priority. There is a huge interest from researchers and scientists to provide many techniques and proposals to assist the medical systems and many have contributed to delivering solutions on how to stop this disease [2]. With the use of technology and science, it has become a lot more comfortable to engage in implementing advanced algorithms to suppress this virus [3]. Screening methods have emerged a lot as it plays an important role for not only travellers but also to the people [4]. These types of methods [5,6] are mostly needed in places like airports, bus stops and hospitals where most people accumulate. Since many detection and screening methods have developed to identify the disease [7,8], it has become possible to quarantine the infected ones thereby reducing the communication between the people. But again, the time to diagnose the disease should be less so that more people can be identified and take treatments as early as possible, which currently is not as effective as it should be. For example, the thermal screening method which has proved to be very helpful because it detects anyone with elevated temperature [9], thus easily recognizing people who are sick. But it is not enough as it detects everyone, which takes up a lot of time, thus delaying the screening process for a large scale of people. The major problem tends to be that, contacts made by the infected people will only help to increase the number of COVID-19 cases [10]. Therefore, smart screening systems [11,12] are needed which consume less time, should be cost-effective, able to track, trace, separate the patients from normal people and most importantly give a high percentage of accuracy to determine whether a person is virus-free or not.

This paper approaches the idea of monitoring and detecting people with the COVID-19 virus through an automated model without the help and intervention of human guidance. Our objective is to make this framework support people working in medical organizations and help identify cases where more people are present. At the beginning of the pandemic period, many researchers have concentrated on identifying this virus by implementing advanced Deep learning algorithms through X-ray images and CT image analysis [[13], [14], [15]], which gave a high accuracy rate. As time went by, many new frameworks and models have been implemented [16], which not only detected but also efficiently improved in tracking the number of cases [17]. Although at the time of writing this paper, vaccines for COVID-19 were not yet made and were still in the making stage, this model is an exception because it solely depends on detecting and identifying the COVID-19 cases, which is necessary even after the vaccines are authorized.

Our paper focuses on aiming and developing a new framework that is automated and works at the real-time approach. Deep Learning techniques like Background Subtraction [18]and Convolutional Neural Networks [19] have been presented as two of our framework's three methods. Thermal Imaging [20] is another method that acts as the first and foremost part of our proposal. A detailed description is explained in this paper's methodology section, which enhances our model's working.

Our proposed framework will include the survey and dependence on research papers due to the temporary lack of infra and platform availability considering the country's COVID-19 situation. We aim at introducing our framework while highlighting the methods that promise the whole process working state. We hope that our reader is aware of these contemporary challenges.

We start with the research gap and related work section, which describes the development of many models, methods and the gap between the past and recent studies and also the research close to our work, done by many researchers and engineers. Section 2 provides a detailed description of the methodology and Section 3 provides the discussion. Section 4 explains our future work directions, followed by concluding remarks and references.

Related work and research gap

Shortly after the start of the COVID-19 outbreak, many countries had declared a state of emergency and declared lockdown in most of the places. The screening tests have started to diagnose the cases to trace and contain the pandemic. Many methods were suggested and implemented to detect the virus as it was necessary to find the patients to prevent further infection. These methods are tested on people mostly travelers who have high chances of being diseased. Airports, being the largest hub for people traveling from different regions, are the first places to begin the screening tests. But countries like India where the population is very huge, screening tests should not be limited to only airports. It should be facilitated to places like bus stops, train stations, medical centers and many more, where people are mass-populated. Generally, screening of people not only means diagnosing the patients but it means that it should be able to track both symptomatic and asymptomatic people to avoid further infection of the virus. Ideal screening tests are required which increases the chances of evading this pandemic. Ideal screening tests are those tests that meet the ideal criteria like [21].

The speed of the testing should be quick enough and the results must be available in a minimum amount of standard time.

The screening should be done on a large scale of people especially in a populous country like India, where it is essential to screen patients to minimize the infection before it rapidly spreads out.

The testing should be cost-effective so that screening could be done at every corner of the region.

The screening of people should be automated to enhance the results.

Along with testing, tracing, and tracking of patients and asymptomatic people should be done.

Most important, screen testing should give results with high accuracy without false negative and false positive output.

Currently, the majority of the tests for COVID-19 are coming from RT-PCR (Real-Time Reverse transcription polymerase chain reaction) which according to the Health Ministry of India is the ‘gold standard’ frontline test for COVID-19. This method is not only preferred in India but also chosen by many countries as of now it is the only technique that can give near to accurate results for detecting the presence of this virus. Although it gives better results than many other techniques, it takes up a great amount of time and requires equipped laboratories and trained technicians to work on this method which makes it an expensive method [22].

Antigen technique is a detection technique which aims at detecting the proteins of the virus called antigens. The strength of this test is that it is faster and easier to perform than the RT-PCR method. A COVID-19 antigen test takes approximately 15–30 min to complete, unlike the RT-PCR which could take days to give the result. It requires very little expertise and it is less expensive as well. Although this method is very good with the speed, it fails to give accuracy as it has a higher chance of showing false negatives thereby negative results do not rule out the infection. These negative results are further ultimately needed to be confirmed with RT-PCR test for accurate results [23].

Serological tests are used to find out the presence of the virus in a body and able to identify people who are previously infected and recovered. But this method does not find the current virus since the antibodies take several days to weeks to develop after the person gets infected and by this time, the infected person has the potential to spread the virus if not quarantined. The reason being is that these tests are not completely developed yet and require more research in this area [24,25].

Thermal screening is another method that detects the high temperature of people thus assuming to have a possible virus. Thermal cameras are inculcated which could scan the people. Though these cameras could sense the elevated temperature, they are not precise enough to tell whether someone has a normal fever or the fever which could be the symptom of this virus [26]. This method also has a high risk of showing false-negative and false positive results which is not suggestible for the screening test [10].

There are many other methods and techniques implemented by different countries to reduce the infection. To enhance the screening tests, smart systems are required. The smart systems should be able to pass the criteria for ideal screening tests which could lead to fewer COVID-19 cases. Along with the smart screening tests, a smart strategy is also required to stop the infection rate from person to person [27]. A three-way TTT strategy is an example of a smart strategy. This strategy is already in use in countries like South Korea and Singapore which follow three steps; Testing, Tracking and Tracing. The main purpose of this strategy is to find and suppress the local outbreaks across the region. This strategy ensures that the infected person coming in contact is obligatory to be traced, tracked, tested and isolated. Although this strategy is successful, it applies to countries with less population since this method requires a large number of people to monitor the infected ones as well as non-infected ones. For a country like India, this strategy may look like a failure but if put to use in a smart way it could lead to a tremendous reduction in the of COVID-19 cases [28].

Recently, many methods have emerged during this pandemic that show an outstanding possibility of avoiding this infection and detecting several COVID-19 cases. These methods have made many improvements to overcome the difficulties faced by the techniques and tests mentioned above. These have emerged well in such a way that classifying and identifying infected people became no more a threat and are still coming with better algorithms. In [29], the authors have proposed a Deep Learning system(COVID-19Net) for COVID-19 diagnostic and prognostic analysis, which involved a hierarchy of Deep Learning neural networks. This model was based on the Dense-Net-like structure and predicted the input patient's probability of being infected with COVID-19.

In [30], a monitoring system is proposed to avoid people who do not follow the government's proper guidelines. Different types of CNN architectures and thermal imaging systems are opted for easily distinguishing people through their body's temperature, reactions and actions. The same approach has been made by [31] aiming to check social distancing between the people. This proposal is made by exploiting YOLOv2 (you look at once) using thermal images. The Street seek project [32] has been developed for a smart city program for analyzing the engagement between the people. This technology uses a Thermal and Deep Learning based platform that is capable of gathering real-time data. In [33], the authors have developed automated thermal monitoring for H1N1 screening. This system primarily focuses on segmentation between the foreground and the thermal image background. After removing the complexity of the thermal image, it successfully suspects the person with the influenza virus.

The authors of [17] have proposed a COVID-19 detection system that would collect real-time symptom data from IoT infrastructure, which eventually monitors both potential and confirmed positive cases. The concept of [34] dynamic spatiotemporal fuzzy background removal from an infrared thermal imaging sensor is implemented where background removal occurs, and the foreground complement algorithm complements only the extraction of the human body.

Our paper's attention also deals with some of the ideas and processes based on the above-mentioned related works that their respective authors have achieved.

Proposed framework

This section explains our framework's methodology and process, which includes different stages that are automated throughout the whole system. Fig. 1 shows the proposed framework of Real-Time Detection and Identification of COVID-19 suspected patients. The essential steps for this workflow are

Fig. 1

Real-Time Detection and Identification of COVID-19 suspected patients.

Capturing people by streaming thermal videos from thermal cameras (infrared cameras), operating as surveillance cameras.

Separating the foreground from the background on the thermal videos passed from the previous stage generates thermal images containing only people in a thermal form.

Applying an architecture of Convolutional Neural Network (CNN) to determine the actual cases of COVID-19.

Finally, the algorithm will decide to quarantine a person or not, thereby making it easier to classify people.

The proposed model consists of mainly three phases, and Fig. 1 presents an abstract view of the proposed model. The first phase is the phase where real-time capturing occurs that takes as an input of streaming the people's thermal videos. This step is detailed more in Section 2.2. The next stage is to extract the people's thermal images from the thermal video sequences captured in the first stage. This phase is explained more in Section 2.3. The next and final step is to apply an architecture, more like a pre-trained model of the type Convolutional Neural Network, which ultimately detects whether a person needs to be isolated or not. This step is described more in Section 2.4. Section 2.1 is an algorithm that represents the structure of the proposed model.

Algorithm

Thermal imaging

Thermal imaging is the detection of the long-wave infrared radiation emitted from objects' surface, both living and non-living beings. The emission of radiation depends upon the body's temperature, and according to it, measurement of that temperature is recorded [35]. In our proposed model's first step, thermal infrared cameras are suggested to visualize the body temperature [30] as a video sequence. The capturing of each and every one is of utmost necessity as the crowd may contain asymptotic people, which prompts us to leave no one. This phase is implemented in a real-time scenario, most significantly in airports, bus stops, or train stations where travelers are assumed to be present. This stage acts as an input stage where a 3-channel thermal sequence is a foundation for the next phase [20].

However, an important point must be considered as surroundings play an essential role, so as a result, additive noise would be present with the slightest environmental conditions [34]. The video sequences may likely have a thermal disturbance, making it challenging to target humans when applied an architecture of Convolutional Neural Network (CNN). Therefore, it is vital to remove these noises before any dynamic complexities occur during the detection and ultimately give false results. A background subtraction or background removal process is acknowledged as an optimal solution for this crisis to acquire a good outcome. A more detailed explanation is present in the following section.

Background subtraction

A standard approach for the background subtraction model is a method to take input as different image frames of a video and initialize a background frame along with subsequent frames moving to get subtracted from the background model. The background model is a reference image representing the frame's background and is updated together with the frame sequences [36]. Fig. 2 is a fundamental approach for background subtraction.

Fig. 2

Schematic Diagram of Background Subtraction.

But, thermal imaging gives a value depending upon the temperature in foreground and background contrast [35]. A right foreground and background contrast will give good results. For this, we need to find the region-of-interest (ROIs) that contain the foreground thermal values. These ROIs are reliable temperature data that extract the local regions of interest from segmented body regions. Here the segmented body region is referred to as people that compute the body region. James W. Davis and Vinay Sharma [37] have introduced a novel approach called Contour Saliency Map (CSM) such that only the foreground gradients are preserved. The CSM computes for each ROI where each pixel's value in the CSM represents the confidence/belief of that pixel belonging to a foreground object's boundary. The CSM only preserves the gradients in the input image that are both strong and significantly different from the background.

Nanhao Gu, Bo Yang, and Tong Zhang [34] have stated that by applying a dynamic spatiotemporal fuzzy background removal algorithm, which could adapt to dramatic appearing or disappearing thermal disturbance as well as the changing environmental temperature indoors. This algorithm explains that we need pixels removed for the human body region after background removal but remained by using the threshold segmentation. We use these pixels to complete the result after background removal thereby extracting a clear human body. We could enhance the algorithm to outdoors as our main objective is to screen the people who are travelling and are more prone to get infected. S. Hunziker, et al. [38] have implemented background subtraction through PCA (principal component analysis). Applying PCA-based background subtraction, spatially and temporarily varying thermal background emission can be removed.

We could implement one of the works mentioned above as all of them have achieved the desired outcome with great accuracy. After the background removal removes the video sequences' thermal disturbances, we are left with the foreground thermal images, which act as an input to the next stage. The output of this stage is the extracted human body images without any non-human background.

Convolutional Neural Network (CNN)

Convolutional Neural Network is the core architecture of Deep Learning on which many pre-trained models have been established and an undoubtedly successful technique that became the root of a wide range of applications. The basic network consists of multiple layers assembled which have different functions of their own that work sequentially to classify the given input. Three principal layers (see Fig. 3 ) are used to build a full Convolutional Neural Network. Fig. 3 shows the basic building blocks of CNN.

Fig. 3

Schematic Diagram of CNN.

Convolutional layer

The extraction of features from the input image is called convolution. It learns image features and works in coordination with pixels by using small squares of input data. Convolution is applied to the input image using a convolution filter to produce feature maps. A convolution filter, usually called kernel or feature detector [39], is a small matrix of numbers that are passed over the input image to transform to a feature map (also called Activation Map or Convolved Feature) obtained after convolution operation. Subsequent feature maps are calculated by the general formula [39]:Where Input Image is denoted by I and kernel as F. The indexes m and n are the rows and columns respectively, of the resultant matrix feature map and also the size (m x n) of the filter. The convolution operator is denoted by the symbol *.

In a typical CNN, in addition to the convolutional operation, usually, a non-linear operation is performed to terminate any existence of negative pixel values by introducing non-linearity. ReLu is one of the activation functions which is used most of the time to perform this operation [40].

Pooling layer

This layer is often used for subsampling which expresses the reduction of dimensionality of individual feature maps obtained from the previous layer. Although there is a decrement in size, it successfully preserves valuable information of the image allowing the network to work efficiently. This layer is preceded by a sequence of convolution layer when the number of parameters is needed to be reduced. There are mainly three techniques of pooling (Max, Average, and Sum) and the most commonly used pooling technique is Max Pooling which considers the largest element from the feature map [40].

Fully connected layer

The output of the Pooling layer becomes the input for the Fully Connected Layers. In this layer, every node in the previous FCC layer is connected to every node in the next FCC layer and every node holds certain information that is necessary to the input. The classification is done based on the features extracted by previous layers (Pooling and Convolutional). At the last FCC layer, an activation function is used to predict the class of the image which is nothing but the classification of the input images. Here, the classification is done on thermal images of people who have a virus and who do not have a virus. These layers use the concept of parameter sharing, where weights are shared by all neurons of a feature map, and local connectivity [41], where neurons are connected to only a small part of the input image. This is the specialty of CNN which increases performance as well as the efficiency of the whole network. The layers mentioned above are the basic norms of CNN as in reality multiple layers are used along with pre-trained models for different types of applications [42].

For detailed information regarding the concept of CNN refer [43]. Since we are dealing with extensive thermal data images, pre-trained networks of CNN would be more satisfying. These pre-trained networks are already trained for another task for which a lot of data is available. Weights of used networks are re-utilized for the new objective [44]. For our data, the ability to discover necessary information in the thermal images is crucial. According to the technique proposed by Zeiler et al. [45], three steps are stated, which are iterated over:

Masking a part of the input image

Classifying by using softmax activation function in the output layer.

The probability corresponding to each class is accumulated in a matrix with the same dimensions as that image.

Another method could be adapted, which can extract the human body's whole segmentation from the thermal images. With this approach, automatic thermal analysis of accurate calculations of pixels is possible. It makes the area around the human body be examined individually depending upon the probable temperature [46]. This method allows the definition of temperatures precisely that would adequately distinguish between infected and non-infected people.

Comparison and output

As mentioned above in the previous section, the classification between the COVID-19 infected and non-infected people can be done in various ways. The most suitable way is to compare with a database having thermal images of already infected people thereby concluding the number of cases of either COVID-19 positive or COVID-19 negative. The people having negative results will be under surveillance and the people having positive results will be quarantined and the authority will be informed about the patient’s status.

Discussion

Related research has been done to support our proposed model's methodology that justifies that this model can be implemented and can have high accuracy.

The work of [34] makes primary progress for us in successfully capturing humans from a dynamically changing background of an infrared image. The experimental results indicate that the dynamic fuzzy background removal (DFBR) could adapt to any indoor environment's thermal disturbances. This algorithm is achieved through infrared thermopile array sensors that ensure to display of human targets in real-time. Similarly, O.Janssens et al. [44] have implemented a method that determines the machine's condition when applied to infrared thermal data. The proposed system can detect the fault in rotating machinery very accurately, acquiring a 95% and 91.67% accuracy rate.

CNNs [46] were involved in multiclass segmentation in thermal infrared face analysis. Trained architectures do the face classification in real-time by displaying the relative temperature from the learned areas, leading towards the detection of infection like COVID-19. Another CNN [47] approach addresses the target identification and recognition in the wild for infrared imaging in defense applications. The experiments show identification scores above 70% for localization and scale errors, which prove to be a better result. M. Irfan Uddin et al. [30] have developed a novel technique that monitors and guides people to distance themselves from infected people. Different CNN models were employed, among which ResNet50 gained a relatable high accuracy.

Aparna Akula et al. [48] have presented a target detection algorithm that uses both spatial and temporal information for detecting targets in infrared thermal imagery. The background subtraction is applied in between the two stages of this model during the temporal processing. This model has been tested and has both high detection rate and false alarm rate. Similarly, Yijun Yan et al. [49] introduced an efficient framework that uses deep neural network-based background subtraction for the detection of pedestrians on thermal video frames. The result has been better with the usage of DNN that allowed the proposed method to detect pedestrians successfully withstanding dark shadow and illumination changes in the video sequence. Nur Syazreen Binti Ardnan [33] has proposed a method to screen H1N1 patients using background subtraction in his thesis.

The above-stated works provide significant results, which can depict that our proposed model has a chance to give better results and can stand among them.

Fig. 4 depicts the screening process of travelers at the airport. It includes thermal scanning through thermal cameras, which sends the data for detecting the virus. The data sent is for the background subtraction method, where it removes the unwanted background focusing only on people. This method generates images for the classification of people who need to be quarantined or not. The travelers diagnosed with the virus are guided to the emergency, whereas virus-free people are informed to the authorities. This figure is just an illustration of the model working at the airport, and it is not limited to the airport, but many public places like bus stops, metro stations, shopping malls and many more.

Fig. 4

Passenger screening process at Airport (Proposed model working).

The aim of the model is based on the notion of overcoming the limitations faced by many screening methods and the main characteristic of this model is that it is Fully Automated as it helps in decreasing the contact between the patients and medicals, thus avoiding the spread of infection. Since it is autonomous and no monitoring is done by officials, this model can assure that no one gets excluded during the screening process leading to minimal human contact.

Comparative results and conclusion

The evaluation criteria for our proposed model states a threshold temperature for each and every person who have to be thermal screened. This threshold temperature is calculated in the process of our proposed methodology and eventually identifies the COVID-19 patient. The comparison has been made [Table 1 ] with respect to the works done as cited in the above sections. The algorithms may have different perspectives but the working outcome is almost same which becomes easy to compare the results. These algorithms can be implemented in our proposed methodology to have better effects of our classification.

Table 1

Results comparison.

Algorithms for Thermal Imaging and Background Subtraction Method	PPV
Background Foreground Segmentation(ROI Extraction) [35]	90%
Deblur-SRRGAN [20]	84%
Target region extraction [48]	98.29%
DNN based background subtraction [49]	89.22%
Contour Saliency Map (CSM) [37]	94.8%

Other important works which satisfies our proposed framework is the dynamic spatiotemporal fuzzy background removal algorithm which identifies thermal human behavior successfully against all the thermal disturbances as well as environmental temperature changes [34]. This work has successfully denoted the working of the background removal and thermal disturbances which is required for further stages of the process. Another works which determine the background removal include integration of background removal technique [36] and PCA based approach for thermal background removal [38]. These different algorithms work on the same goal, to remove thermal background ultimately removing thermal disturbances. These works are vital as these determine the thermal images of the public which are needed for further classification of COVID-19 cases through multiple CNN methodologies. Implementing and classifying through CNNs is relatively easier than segmenting the thermal information of inputs. These related works achieve the results of our proposed framework which satisfies our conclusion.

The aim of the proposed model is emphasized in the areas where more people are concentrated like metro stations, airports, medical centers, shopping malls and public places. The chances of not getting this virus in those regions are approximately zero and therefore the number of COVID cases tends to increase exponentially. The fundamental priority is that no single positive case goes unidentified, hence the need for the detection of the Covid-19 virus is very essential and if the proposed model gets adopted, a high percentage of accuracy could be achieved as compared to other models and therefore could be suitably applied for detecting and reducing the number of Covid-19 cases as much as possible. It is also hoped that this research model will aid in the development of medical assistance throughout this pandemic. Our automated system can produce an outcome with accurate results if implemented.

Although the framework's methods may be referred from the corresponding technique's works, the overall proposal is novel to the best of our knowledge. Our intention towards this approach is to share the idea in favor of enhancing biomedical technology and prioritizing people's healthcare.

Future work

Future studies include sanitization through IoT and ultrasonic soundwaves for the permanent prevention of Covid-19 transmission. The crux of this concept is that when the virus is abundant in the atmosphere it can cause possible infection to the people having weak immunity. To prevent this, treatment with UV rays could be done where only the virus gets eliminated without actually harming the skin of the person. We plan to develop and implement this approach for a better sanitization.

CRediT authorship contribution statement

Bharati Peddinti: Writing - original draft. Amir Shaikh: Resources. Bhavya K.R.: Conceptualization. Nithin Kumar K.C.: Supervision, Writing - review & editing.