Internet of Things with Deep Learning-Based Face Recognition Approach for Authentication in Control Medical Systems.

Internet of Things (IoT) with deep learning (DL) is drastically growing and plays a significant role in many applications, including medical and healthcare systems. It can help users in this field get an advantage in terms of enhanced touchless authentication, especially in spreading infectious diseases like coronavirus disease 2019 (COVID-19). Even though there is a number of available security systems, they suffer from one or more of issues, such as identity fraud, loss of keys and passwords, or spreading diseases through touch authentication tools. To overcome these issues, IoT-based intelligent control medical authentication systems using DL models are proposed to enhance the security factor of medical and healthcare places effectively. This work applies IoT with DL models to recognize human faces for authentication in smart control medical systems. We use Raspberry Pi (RPi) because it has low cost and acts as the main controller in this system. The installation of a smart control system using general-purpose input/output (GPIO) pins of RPi also enhanced the antitheft for smart locks, and the RPi is connected to smart doors. For user authentication, a camera module is used to capture the face image and compare them with database images for getting access. The proposed approach performs face detection using the Haar cascade techniques, while for face recognition, the system comprises the following steps. The first step is the facial feature extraction step, which is done using the pretrained CNN models (ResNet-50 and VGG-16) along with linear binary pattern histogram (LBPH) algorithm. The second step is the classification step which can be done using a support vector machine (SVM) classifier. Only classified face as genuine leads to unlock the door; otherwise, the door is locked, and the system sends a notification email to the home/medical place with detected face images and stores the detected person name and time information on the SQL database. The comparative study of this work shows that the approach achieved 99.56% accuracy compared with some different related methods.

1. Introduction

In the world of technology, security has become a necessity in everyday life. Nowadays, technology is becoming an integral part of everyone's lives, so the security of every home is not left behind [1]. With the latest development in the area of artificial intelligence and big data, there is a huge gap for advancement in the field of computer vision for face recognition systems, especially in medical and healthcare environments and places to combat the spreading of infectious diseases due to touch authentication tools. In this work, the instance of a smart door unlocks the system. We have to remove conventional features and update the system to remodel the device. The main issues of a traditional security system are that anyone can access the door by copying or robbing the key and breaching the pattern. We can just update this conventional lock system into a smart one to remove such drawbacks. Face recognition technology is one of the hottest topics in computer vision and biometrics systems [2], since it is a challenging task to recognize faces with distinct expressions. There are many algorithms to run the face recognition model. We take the pretrained convolutional neural network (ResNet-50 and VGG-16) for feature extraction and compared with LBPH algorithm, and for classification, we used SVM algorithm. The ResNet-50 gives the best result compared to other algorithms [3] for image recognition. And Haar cascade technique is used for face detection purposes. We use Haar cascade classifier due to its high detection accuracy, speed, and low false rate. It was trained a lot of positive (face) and negative images (without face) [4].

The system's drawbacks are blindly trusting someone or in traditional security measures like a key, ID card, password, or pattern and giving permission to the system. Despite that, such types of security systems have weaknesses. For example, password forgot losing the key or being stolen from an unofficial person [5, 6]. Therefore, we need to remove the conventional security measures and update the new ones. So, the face recognition (FR) is the most desired method of biometric technology system [7, 8]. In comparison to other smart technology, like voice recognition, retina scan, and fingerprint, face recognition will be considered more natural. FR can only give privileges to certain people. Therefore, we used ResNet-50 and VGG-16 for deep feature extraction, and for classification, we used SVM algorithm and then compared the result with LBPH algorithms for FR-based door unlock system. This system contains a webcam to detect the face images of a person and then validate it with the images in database. Once the face is successfully recognized, the door lock opened and send an email to the homeowner along with the face image and save the information to the SQL database; else, the door remained locked and also notify the owner through email alert as an unknown person with detected face image. The main controller in this system is Raspberry Pi 4 and the relay circuit. The FR system authenticates the person's identification with the database system. We adopt the (open-source computer vision) OpenCV library. The flow of the FR system is equipped with the camera and relay module solenoid lock, and Haar cascade classifiers were used for the face detection process. After the face detection process, the system will compare and classify the detected face images with the database images; this classification is done with a SVM algorithm.

Furthermore, there are many academic efforts to find out the robust and reliable dataset for testing and formulating the proposed study: getting a better dataset is an important task especially for facial recognition. To find the effectiveness of the method, accurate datasets are needed which (1) contain a large number of person face images and (2) check the effectiveness of these methods. Our contributions in this study are as follows:

IoT with DL can help users in this field to get an advantage in terms of enhanced touchless authentication, especially in spreading infectious diseases like coronavirus disease 2019 (COVID-19)

Even though there is a number of available security systems, they suffer from one or more of issues, such as identity fraud, loss of keys and passwords, or spreading diseases through touch authentication tools. To overcome these issues, IoT-based intelligent control medical authentication systems using DL models are proposed to enhance the security factor of medical and healthcare places effectively

This work applies IoT with DL models to recognize human faces for authentication in smart control medical systems. We use Raspberry Pi (RPi) because it has low cost and acts as the main controller in these systems

The installation of a smart control system using general-purpose input/output (GPIO) pins of RPi also enhanced the antitheft for smart locks, and the RPi is connected to smart doors

We analyze and compare several in-depth learning methods according to the architecture implemented and their performance assessment metrics

In this study, we use novel approach using Raspberry Pi-based cloud-assisted framework for unauthorized person detection and recognition and send the detected person image to the owners for better safety

The Viola-Jones detection classifier is used in the proposed study, which provides efficient detection results; furthermore, the Viola-Jones Haar cascade classifier is computationally inexpensive and it is suitable for Raspberry Pi-based framework

The DL methods (ResNet-50 and VGG-16) are used for deep feature extraction and then used its outputs to feed the SVM classifier. The pretrained models are used as fixed feature extraction when the dataset is small

The training samples for various face poses, occlusion, and illumination are generally required. Sometimes it is difficult, if it is not possible due to restriction of time and space constraints; it is difficult to obtain sufficient facial image dataset. To address the issue of insufficient samples, effective data augmentation techniques are used in [9, 10]. The idea behind the augmentation is to generate virtual sample to increase the training datasets. In this study, geometric modification, image brightness changes, and performance using different filtering techniques are used to modify the training datasets.

This proposed system is easy to install with low operational cost. This system plays a major contribution to the field of smart medical control system.

2. Related Work

This part gives different techniques for door unlock system. The preceding work is where we deal with algorithms and used different electronic devices for door unlock system.

The author in [11] proposed a “real-time control using Arduino-based door unlocking system”; in this study, they used radio frequency identification (RFID) codes to scan card for door unlock. If a person wants to enter the room first, it is needed to scan the card then someone can able to enter the room. The drawback of this system is misplaced a card that leads to access the door. The author in [12] proposed a “secured room access module”. In this work, he used a keyboard-based unlock system using a microcontroller where the user needs to enter the password to get access to the door. This method is reliable if we compared with the previous study and also better in today's era. Although this method is better, but there are some drawbacks; if someone notices your password, then someone can gain access to the door. The authors in [13] used a system that takes images through Raspberry Pi 3 model B and compared the detected images with database images, although the confinement in this model did not work properly due to poor lighting. The authors in [14] proposed an IoT-based security system using face recognition. In this system, Raspberry Pi is used along with camera module to take images and compared it with database images. It used OpenCV with Python for feature extraction. This system performs better for both known and unknown images. The authors in [15] have proposed door lock unlock system using web application. In this work, they monitor the door for locking and unlocking using web application. Someone can get access with his mobile and also check the status of door locking/unlocking. The drawback is someone hack the code and easily enter the door. The authors in [16] proposed an embedded platform. In this study, taking image in embedded system with Raspberry Pi considers the recognition and image capturing requirements. Raspberry Pi with other hardware device develops on this platform. The designed system is fast to run image recognition algorithm. The data flows in between camera module and Raspberry Pi. The authors in [17] proposed face detection along with equivalent time recognizer using OpenCV and eigenface; this proposed system has done on Raspbian operating system in Raspberry Pi model B. Pi camera is used for image capturing with the help of face recognition ability. This method assists the police tasks to recognize detected faces from captured images of a camera-based system, which can be immensely a reliable system. The authors in [18] proposed the inputs for the project are capturing the face in camera module from webcam and classifying on that image using CNN algorithm then verifying the user. The author in [19] has implemented the work based on Raspberry Pi model B, with Pi camera for capturing the images and then converting the image into grayscale. He finalized that the results are good and the system is technically advanced compared to image interface on a private PC. The authors in [20] have proposed door locking/unlocking system through different controllers. They used technology like Bluetooth that is available on gadget and consumes less power. They also added special features to enhance the security capabilities and make it easy to use. The authors in [21] proposed the global system for mobile communication (GSM) based on digital security system. This proposed system has designed built-in near-field communication (NFC) capabilities and this would become a key for door lock/unlock using the logical link control protocol that matches the password to lock/unlock the door. The authors [22] proposed a face recognition-based real-time application, by generating a MATLAB code using an image acquisition tool box, with the simple approach that is principle component analysis (PCA) and eigenfaces. The authors in [23] proposed face recognition-based real-time security system using Raspberry Pi. This work has done using LBPH, fisherfaces, and eigenface algorithms. To compare with different processors and know the accuracy and time complexity of the model, this system is secure and no one can get access without the face matching. Recently, a researcher published a number of research papers using deep neural networks in the field of facial biometrics with impressive results, and it compared with traditional algorithms [24]; for facial recognition, CNNs are trained using a data-driven network architecture. In addition, CNN combines the extracted features and classifier into a single framework [25]. A CNN model mainly includes convolutional layers, pooling layers, fully connected layers, and an input and an output layer. CNNs are better for feature extraction and play an important breakthrough for face recognition.

There are multiple door locking/unlocking systems have been designed. The comparative study of door unlocking system based on face authentication and verification is depicted in Table 1. The table comprises of different types of smart door locking/unlocking systems that have already designed in previous studies. These studies mainly require the additional device for system operation and obtain the system authentication condition using smartphone; to operate the system, user needs to input command manually through smartphone interface. The problem arises in the systems, if the person is not familiar to the smart technology and its user interface, which leads the difficulty to use. Therefore, this conducted study provides an efficient way, which is not only for safety purposes for real-time face detection and recognition technique but also easier to use, because the design of the system is more user friendly, which does not need any extra manual commands nor technical skills are needed. The entire system is embedded into a door that is automatically operatable. The innovation of smart door locking/unlocking system will also be discussed in this study which is depicted in Table 1.

Table 1

Related work.

Verification technique	Adopted device	Input	Reference no.
Password verification	Keypad	Needed	[26]
Near-field communication (NFC) verification and password comparison	Android phone	Needed	[21]
Smart phone-based network authentication	Android phone	Needed	[27]
Taking images for face recognition authentication	Android phone	Needed	[28]
Real-time face recognition	Not needed	Not needed	Proposed method

3. System Architecture and Design

3.1. System Architecture

This proposed system has many modules such as image module, control module, and door locking/unlocking module shown in Figure 1. The image module takes a face image of a person and sends it to the main control system (Raspberry Pi) for further process. It is achieved through a web camera (Logitech). The door locking/unlocking module mainly contains 5 V relay circuit (RC) and electromagnetic solenoid lock (EMSL), which deal with locking/unlocking the door, are discussed in [29, 30]. The control module is heart of the system, which is realized by utilizing the Raspberry Pi 4 module B+ depicted in Figure 1.These particular system responsibilities include taking face images through web camera, process the image as required, contain the facial image database, compare obtained images with stored database image, and send query to the door locking/unlocking module. The function of the control module acts as a web server for sending and receiving the emails notifications. For short message service (SMS) notifications and backend access have been studied in [31].

Figure 1

Block diagram architecture.

3.2. System Description

3.2.1. Image Module

This module utilizes a webcam (Logitech); the reason of using Logitech instead of Pi camera is due to the effectiveness of cost. The feature of Logitech is high-quality image resolution of 1080 pixels with 30 frames per second (30 fps). It is also good for low-light picture and equipped with night version footage. The interface of camera with Raspberry Pi via USB 2.0 ports that are conducted taking image is presented in Figure 2.

Figure 2

USB web camera.

3.2.2. Raspberry Pi Control Module

The control module of the designed system using Raspberry Pi 4 model B+ is developed and designed by Raspberry Pi Foundation. The feature of Pi 4 contains a 64-bit ARM Cortex A72 4GB of RAM. It has video core VI graphical processing unit (GPU) for the graphical processing application (GPA). Furthermore, it consists of two USB ports and 40 GPIO pins to connect Pi with external electronic devices; the door locking/unlocking module utilized the GPIO pins. Raspberry Pi is developed to execute Linux-based operating system (LOS) having its own operating system, i.e., Raspbian operating system (ROS), and used Python as official programming language. The core module plays an effective role which is responsible for several functions (i.e., obtains image from webcam, processes and stores images, and maintains database image of authorized persons). It detects and recognizes the person which is authorized or not. The control system is responsible for sending the query to lock/unlock the door using the Python programming code through GPIO pins to the relay as shown in Figure 3.

Figure 3

Raspberry Pi 4 model B+ control module.

3.2.3. Embedded Server and IoT for E-mail Notification

The system has another core module which acts as an embedded web server. The main function of the server includes sending visitors detected face image via email to the owner. In this proposed work, the system will notify the homeowner about the detected person using simple mail transfer protocol (SMTP). By using this protocol, the system sends email along with detected face image, either it is known or unknown person to the owner which is shown in Figures 4(a) and 4(b). This system saves the time and name of detected person either recognized or not recognized person in the SQL database file shown in Figure 4(c).

Figure 4

E-mail notification.

3.2.4. Face Detection

There are various face detection algorithms used for different applications, like security surveillance, gaming, and human-computer interaction [32]. The face detection function detects faces from photos or videos and it differentiates from other objects. Viola and Jones developed an object detection algorithm based on Haar cascade classifier [33]. It is machine learning algorithm in which there are lots of positive and negative images used to train the classifier; the cascade classifier is trained for feature extraction and then used for face detection [34]. The use of the classifier is due to its high detection accuracy, speed, and its low false positive rate [35]. The image value of each simple feature is calculated by Paul Viola and Michael Jones methods [36] which can be calculated by using the following equation: where n denotes the number of pixel value and F(x) is the value got from the image, which is depicted in Figures 5 and 6.

Figure 5

Haar features.

Figure 6

Feature extraction.

There are many detection algorithms that have been developed, with various computational times and detection methods. Paul Viola and Michael Jones developed a Haar classifier which can boost rejection cascade architecture. This architecture was used for the first time in a real-time face detector with a high classification accuracy [37]. Paul Viola and Michael Jones were able to produce rapid facial detection by developing a method to quickly calculate digital image features. The Haar features are good at detecting edges and lines, and it is effective for face detection. If the edge and line features give to the network or any algorithm that detects faces, then it is only able to detect objects with clear edges and lines.

The method proposed by Viola and Joins [38] was improved later upon the work presented by Lienhart and Maydt [39]. This method has a high detection rate at the cost of low rejection [40]. With a low rejection rate, the number of false positive or wrongly detected objects can be high. By applying image processing and multiple “feature” classifiers to the area of the image detected containing an object, we can lower the false positive rate. With only a slight increase in computation time, the OpenCV can also provide a set of public domain classifiers that can be used to improve the false positive detection rate of a single classifier. Using the baseline classifier, we determine areas of the image that have a high probability containing a face. These areas are used to create regions of interest for the multiclassifier system.

3.2.5. Face Recognition (FR)

The FR model basically finds out the identity of the face image by comparing the images stored in database studied in [41]. In general, the FR is divided into three main stages, i.e., face detection, feature extraction, and FR, presented in [42]. The face detection model is used to locate the occurrence of face in the image. In some libraries, face alignment is used to enhance the potency of FR models; in general, there are some difficulties in recognizing the same faces with different orientations. The feature extraction is done on the aligned face to obtain main features which is useful for recognition. The face patch detected is transform into vector points depending on the implemented model. Lastly, FR is carried out by feature comparison from the detected face-to-face features stored in database utilizing classification model based on confidence score. Figure 7 explains the whole work flow of FR system.

Figure 7

Face recognition work flow.

3.2.6. Pose

To address the problem of pose variation, we can use a multiview face visual approach, using both frontal face recognition and profile to capture the entire face pose. Therefore, we perform some preprocessing steps to detect the faces, like (1) setting the face image size and (2) converting the image into grayscale. (3) The illumination is normalized using a histogram equalization algorithm, which are shown in Figure 8.

Figure 8

The detected frontal and profile faces.

3.2.7. Illumination

The illumination represents light variations. A slight change in lighting conditions creates a major challenge for automatic facial recognition and can have a profound effect on its results. If the illumination tends to vary, the same individual gets captured with the same sensor and with an almost identical facial expression and pose; the results that emerge may appear quite different. Illumination changes the appearance of the face drastically. It was found that the difference between two identical faces with different illuminations is higher than two different faces taken under the same illumination which is shown in Figure 9.

Figure 9

The illumination variation.

3.3. Proposed Methodology

3.3.1. Face Image Dataset

We have collected 8422 face images of 100 different people in the RGB format shown in Figure 8; these images are captured through camera, and faces are automatically cropped using face OpenCV library. The split rate is 70% training, 15% testing, and 15% validation, where each category has 5895 training images, 1263 for validation, and 1263 for testing images. Each image has different sizes and backgrounds. The images are taken in different places and different lightening conditions for better recognition accuracy. Most existing studies are exploring their method to benchmark databases accrued in a written and managed environment. It has been shown in available research that after experimenting with the face inside in the wild, the accuracy of detection decreases drastically [43, 44]. Our work right here makes use of a small dataset that proves that the proposed study is handled well with real-world applications. The samples of our database images are shown in Figure 10.

Figure 10

Database face images.

3.4. Data Preprocessing

Image preprocessing reduces the processing time and enhances the chances of the perfect matching. Face images are preprocessed to meet the requirements of feature extraction. The steps we used for preprocessing are as follows.

3.4.1. Image Cropping

The location of the image where the face can be found is cropping and can be used for face recognition.

3.4.2. Image Resize

Different image sizes give different information so the best image size needs to be considered in detail. The reason for image resizing is to produce a lower data size, which speeds up the processing time.

3.4.3. Changing the Brightness

Changing the brightness is one of the simplest preprocessing techniques. It refers to the total light or darkness of an image. To increase the brightness, a certain fixed value must be added to each pixel. To reduce brightness, certain fixed values must be extracted from each pixel. The brightness value usually lies from -255 to +255, the negative values refer to the darker side of the image, and the positive value refers to the brighter side of the image.

3.4.4. Converting Color Image to Grayscale Image

Preprocessing of grayscale is also very fast. If we assume that processing a color image of three channels takes three times longer than processing a gray image, so we can save the processing time by removing color channels. In fact, color channel enhances the complexity of the model and often delays processing.

3.4.5. Normalization

It is an important step of data preprocessing in which we have scaled the data values in a specified range (-1.0 to 1.0 or 0.0 to 1.0).

3.5. Pretrained CNN Model

In this proposed method, we used two pretrained convolutional neural network (ResNet-50 and VGG-16) models. These models were used for feature extraction and used them in the classification part.

3.5.1. ResNet-50 Model with SVM

ResNet means deep residual network [45], developed by He et al. It is an advanced convolutional neural network in terms for image recognition. ResNet won the ImageNet competition in 2015 (ILSVRC-15). In this study, we will use ResNet-50 for feature extraction which consists of 5 convolutional layers; this model used an input image size of 224 × 224 × 3 in an RGB format.

This model is comprised in two parts. The first part is for image feature extraction part and second is the classification part which is shown in Figure 11. The implementation process of ResNet model is first we divide the data into 70% for training and 30% for testing dataset. In the preprocessing step, the ResNet-50 network can only process RGB format images which are 224 × 224 × 3. Deep layers were used because of higher level image feature extraction for better recognition task. Before the classification part, we use fully connected layer which is named as Fc-100 that is used for feature extraction by using the activation function. Once we get the feature, then these features are used to train and test the SVM classifier. On the base of these features, the SVM classifier classifies the images either known or unknown faces shown in Figure 9 which show that the ResNet-50 model obtained a higher recognition accuracy.

Figure 11

ResNet-50 model with SVM.

ResNet-50 and VGG-16 are the deep architectures of convolutional neural networks for images. Although ResNet is much deeper as compared to VGG-16, the model size is much smaller due to the usage of global average pooling rather than fully connected layers. This reduces the model size down to 102 MB for ResNet-50. ResNet-50 is superior due to using deep layers for higher-level features which give better distinct features for recognition tasks. Deep neural networks (DNN) improve accuracy and performance. This is because of adding more layers; for that, these layers continue to learn complex features.ResNet-50 is faster due to the bottleneck blocks. It is composed of five convolutional blocks with shortcuts added between layers. The last convolution layer is used to extract deep residual features (DRF).

3.5.2. Pretrained VGG-16 Model with SVM

VGG-16 is a pretrained convolutional neural network model, which can be used for deep feature extraction from images [46], and for the classification, we used SVM classifier. The VGG-16 used in this paper is a 16-layer deep convolutional network and the specific parameters. This network is improved on the basis of AlexNet [47], replacing the 7 × 7 and the 5 × 5 convolution kernels in AlexNet with three 3 × 3 convolution kernels and two 3 × 3 convolution kernels, respectively, whose depth is improved under the same receptive field conditions, thereby the effect of it has been improved.

VGG-16 is basically a transfer learning model and is trained on lots of face datasets, and the input image size of image is 224∗224∗3. These input images are fed through different layers of convolutions and pooling and are followed by fully connected (FC) layers. Each convolutional used filter a size of 3∗3 and stride of 1 and used the same padding. Although the max pool layer used the 2∗2 filter and stride of 2 and the fully connected layers, we extract the features from the last fully connected layer named as fc-100 which is used to extract the features and passed it to SVM classifier to classify the images either known or unknown face. In addition, the hidden layers used nonlinear rectification (ReLU) to show nonlinearity in the system. We employed the generic model of VGG-16 presented in Figure 12. This network consists of 138 million parameters approximately.

Figure 12

VGG-16 model with SVM.

3.5.3. Support Vector Machines

Support vector machine (SVM) is a machine learning algorithm that is used for classification and regressing tasks. It used supervised learning to group the data into two categories. It is trained with a collection of categorized data. The purpose of SVM is to examine which category a new data point belongs to. It cannot only classify the data but also draw a boundary between two categories.

It is one of the important machine learning classifiers which can be used for classification problems. The reason for its widespread use is its simplicity and ease of use. The reason we can choose the SVM classifier is that its performance on small devices with limited resources is implacable. This classifier can be used by many researchers and proved its significance over the other classifier [48]. The other variant of SVM is like fuzzy SVM and LS-SVM, but the performance regarding the accuracy of the original SVM is very good. Many researchers used SVM as a classifier like in medical imaging classification [49-51], fruit classification [52], and face recognition [53]. We use SVM due to its simplicity to implement on Raspberry Pi. The resource available on the raspberry pi is very limited, so the classification process took much time and effective user experience; this proves that SVM is very efficient and compared the performance with a different model with SVM on Raspberry Pi.

3.5.4. Rectified Linear Unit (ReLU)

ReLU stands for rectified linear unit for a nonlinear operation. The output of function is as follows:

ReLU purpose is to introduce nonlinearity in our ConvNet. There are many nonlinear functions such as tanh, sigmoid, and ReLU that can achieve better performance than other functions. In our ConvNet, we use ReLU function to deal with the non-linearity problem. The ReLU operation is shown in Figure 13.

Figure 13

ReLU operation.

3.5.5. System Work Flow

From Figure 14, first start the model, and then it loads the camera module; once the camera is loading, if the person stands in front of the camera, the camera gets the real timeframe from images or videos. Then, load the pretrained convolutional neural network models (ResNet-50 and VGG-16) for feature extraction and classify the images on the base of database images. The classification is done using SVM classifier on the base of confidence scores (CS); we set the CS as 90%. If the CS is greater than or equal to 90%, the face is recognized and door is opened; else, if CS is less than 90%, the door remained locked. Then, the system sends alert email notification to the owner along with detected face images and also saves name and time information on the SQL database which is shown in Figure 14. The system is controlled by the Raspberry Pi 4 model B+.

Figure 14

System work flow.

3.6. Linear Binary Pattern Histogram Algorithm

The open-source computer vision (OpenCV) library gives varieties of feature extraction and recognition algorithms. So, the generally used algorithms are eigenfaces, fisherfaces, and local binary pattern histogram (LBPH) [54]. The LBPH algorithm works better as compared to other two algorithms; it is because it can not only see the frontal face but also recognize the side face which is more flexible. The computational complexity of LBPH is low, but has some vulnerable features such as rotational fluctuations, grayscale variability face analysis, and recognition [55]. These features make LBPH useful for low computation platforms in Raspberry Pi. The computation of LBPH algorithm contains a set of rule that converts image into a decimal value; the conversion process is highlighted in Figure 15. The image is divided into 9-grid (3∗3) pixels; each pixel value range is from 0 to 255. The pixel value can be converted into binary value, before converting into a decimal value in which the center pixel value becomes the threshold. If the pixel value is equal or greater than threshold, the binary value is 1; else, binary value is 0. After that, the system has threshold result of linear binary pattern (LBP), though the resulted threshold value can be converted into histogram by multiplying the threshold result to binary and decimal as depicted in Figure 16, which shows the final histogram to divide the image into 8∗8 grid. Each histogram grid contains a decimal value from 0 to 255. After that, the final histogram is combined which gets the result from each generated histogram. The value of LBPH can represent in a histogram as shown in the following equation: where s represents the sampling point and r is the radius.

Figure 15

Converting grayscale to decimal.

Figure 16

Local binary pattern histogram for face description.

Once the histogram of each input image is obtained, then each histogram represents each image from the training dataset. The face recognition process is performed by feeding a new image, which is generated using a face detection algorithm from a real-time video, which creates a histogram that displays the image and calculates the distance between two histograms with a trained dataset. Hence, the LBPH algorithm used Euclidean distance (ED) presented in the following equation:

The output of this formula shows the image ID with the nearest histogram. The LBPH algorithm also returns the distance that can be represented as confidence score (CS). The Euclidean calculates the gap between images; the smaller the distance has more similarity studied in [56].

3.7. LBPH Work Flow

The flow chart of our proposed method (LBPH) algorithm focuses on several factors, face detection, and recognition for person identification from real-time video source; the proposed system can be discussed using the flow chart which is shown in Figure 17. The proposed activities are explained as follows:

Figure 17

Flow chart of LBPH algorithm.

Camera module used to take pictures from real-time source

Take a photo and save it and compare it to a database image

On the base of confidence score (CS), if the CS is greater than or equal to 90, the face is recognized and door is opened. Else, if CS is less than 90, the face is not recognized and door is remained locked

System sends email to the owner with detected face image and saves the name and time of the detected person in the SQL database which is shown in Figure 17

The interaction between the camera module and the solenoid lock is controlled by a Raspberry Pi using a relay module. The LBPH flow chart algorithm is shown in Figure 17.

4. Results and Discussion

This section describes the implementation result achieved from system development, and model successfully deployed in the container which is able to run on IoT. The system can be trained using appropriate datasets, which is able to detect the face of the visitor/visitors in real-time video frames. The success rate that was observed during the testing accuracy is 99.56% for person recognition, it can decide the authenticity of the person, and the decision made by the system can be used for multiple use cases, for instance, to open security doors based on the authenticity of the person.

The proposed system is compared with pretrained CNN model (ResNet-50 and VGG-16) and LBPH algorithm. We use multicore system for comparing the accuracy along with the model recognition time to run this algorithm on these systems (Raspberry Pi 4 model B+ and Intel Core i7) which are shown in Table 2.

Table 2

Machine for testing model.

Machine	Operation system	Processor	Main memory
Intel	Windows 10	Core i7 (6th gen)	8 GB
Raspberry Pi 4	Raspbian	ARM Cortex A72	4 GB

Figure 18 shows the testing time of pretrained CNN model (ResNet-50 and VGG-16) and LBPH algorithm with different system configurations. So, the LPBH takes more time as compared to ResNet-50 and VGG-16, because LBPH works on a bit-by-bit binary pattern calculation. So, the testing time graph is shown in Figure 18.

Figure 18

Testing time graph of pretrained CNN and LBPH model.

Figure 19 shows the recognition time for each recognition person with time.

Figure 19

Response time for recognition of a person.

5. Experimental Results

In this study, we test the model with different epochs and different batch sizes using Adam optimizer, until we get a good result which is shown in Figure 20. In this experiment, the ResNet-50 model achieves the highest accuracy which is 99.56%, VGG-16 gets 98.49%, and LBPH algorithm achieves 98.47%. The accuracy criterion was used to calculate the performance of our proposed method. The numerical test results and training time are shown in Table 3. The network ResNet with SVM achieves better result as compared to other two algorithms which is shown in Table 3.

Figure 20

Model accuracy.

Table 3

Model performance.

Method	Training time	Accuracy
ResNet-50	1.205 hours	99.56%
VGG-16	1.346 hours	98.49%
LBPH	0.189 hours	98.47%

The main goal of all machine learning algorithms is to decrease the loss. Loss function is also called cost function which is presented in the following equation: where n is the batch size.

A loss function defines how “good” or “bad” our model. The smaller the loss, the better the model and you need to be careful not to overfit the model.

In our proposed study, the training loss is quiet stable with ResNet-50 (ResNet − 50 = 0.012%) model as compared to other two models (VGG − 16 = 0.024% and LBPH = 0.045%) which is shown in Figure 21.

Figure 21

Model loss.

The comparative study of different face recognition methods with the proposed study is presented in Table 4.

Table 4

A comparative study of different methods.

Method	Face recognition accuracy
Eigenfaces [57]	36.19%
HOG [58]	66.45%
Laplacian faces [59]	65.07%
Open face [60]	75.72%
Dense U-Nets [61]	81.43%
Retina face [62]	83.87%
Hierarchical network(ResNet101+FaceNet) [63]	87.36%
ResNet-50, VGG-16, and LBPH (our model)	99.56%, 98.49%, 98.47%

We have successfully programmed with Raspberry Pi and get better results depicted in Figure 22, which shows that a face recognition system using the Raspberry Pi (RPi) has successfully implemented. RPi enabled a 5 V relay adapter. Raspbian was accessed remotely from PuTTY software installed on the system.

Figure 22

Model recognizing faces.

Our model is succeeded in detecting and recognizing the face of the person along with the labelled name, and a confident score of the person was displayed. The process of identifying and recognizing persons faces is performed in a real-time of capturing video frames.

6. Conclusion

In this proposed project, we have successfully implemented a security system that automatically unlocks the face system using the Raspberry Pi 3 model B+. Unlike the traditional method like keys, pattern, or password-based, we get rid of the conventional systems. The implementation of a face recognition system using Raspberry Pi can make it smaller, lighter with low-power consumption, so it is more convenient than a PC-based system. The face recognition system is able to recognize the known and unknown person based on the database images, and the system is also able to send notification emails to the owner with detected face image; this will be done through SMTP protocol. This work has a large room to improve the performance of the image processing component due to the use of the Raspberry Pi module; the processing time took longer; with the help of another method, this work may be modified in a better way.

We worked with algorithms such as Viola and Jones for face detection and for feature extraction; we used pretrained CNN models, i.e., ResNet-50 and VGG-16 with SVM classifier, for classification purpose along with LBPH algorithm for face recognition. The results of these were compared, and we found ResNet-50 worked more effectively by giving 99.56% recognition accuracy as compared to other two algorithms. So, we found that ResNet-50 performed better with the lower computation time as compared to VGG-16 and LBPH algorithms.