Automated Detection of Rehabilitation Exercise by Stroke Patients Using 3-Layer CNN-LSTM Model.

According to statistics, stroke is the second or third leading cause of death and adult disability. Stroke causes losing control of the motor function, paralysis of body parts, and severe back pain for which a physiotherapist employs many therapies to restore the mobility needs of everyday life. This research article presents an automated approach to detect different therapy exercises performed by stroke patients during rehabilitation. The detection of rehabilitation exercise is a complex area of human activity recognition (HAR). Due to numerous achievements and increasing popularity of deep learning (DL) techniques, in this research article a DL model that combines convolutional neural network (CNN) and long short-term memory (LSTM) is proposed and is named as 3-Layer CNN-LSTM model. The dataset is collected through RGB (red, green, and blue) camera under the supervision of a physiotherapist, which is resized in the preprocessing stage. The 3-layer CNN-LSTM model takes preprocessed data at the convolutional layer. The convolutional layer extracts useful features from input data. The extracted features are then processed by adjusting weights through fully connected (FC) layers. The FC layers are followed by the LSTM layer. The LSTM layer further processes this data to learn its spatial and temporal dynamics. For comparison, we trained CNN model over the prescribed dataset and achieved 89.9% accuracy. The conducted experimental examination shows that the 3-Layer CNN-LSTM outperforms CNN and KNN algorithm and achieved 91.3% accuracy.

1. Introduction

Stroke is a worldwide healthcare problem which causes due to heart failure or malfunctioning of blood vessels. It is a common, dangerous, and disabling health disease that affects people all around the world. Stroke is the second or third leading cause of death in most regions, as well as one of the leading causes of acquired adult disability [1]. Over the next couple of decades, the frequency of stroke-related burden is predicted to rise. Stroke causes losing control of the motor function, incoordination or paralysis of all body parts, and severe back pain. Due to stroke, patients will have muscle and neurological trauma and disorders such as cerebrum paralysis [2], trauma and paralytic injury [3], posttraumatic stiffness [4], congenital deformity [5], and Guillain–barre syndrome [6]. Injuries to the cervical spinal cord usually result in loosened leg and arms functions where hip flexors and legs are degraded by lumbar and spinal cord injuries. The survivors of a stroke have a similar condition since they must relearn the lost skills when their brain is hit by a stroke.

A physiotherapist employs many therapies, including nerve reeducation, task coaching, and muscle strengthening to restore the mobility needs of everyday life. Different physiotherapy and rehabilitation programs are needed to restore the function of the upper extremity and increase their quality of life. Some exercises such as motor training (movement exercise), mobility training (restriction-induced), motion therapy (flow therapy), and repetitive task training (workout training) are very effective for learning and taking control of the body [7]. Both for upper and lower limbs, balancing exercises are of considerable benefit to increase the balance after spinal cord injury. It is obvious that serious health problems may lead to death or acquired physical impairment due to injury to the backbone. Different neuroplastic results have shown that it can be recovered partly through adequate rehabilitation exercises [8].

Motor function controls mobility and muscle movement and is a commonly recognized impairment due to stroke. To reestablish motor function, the most important technique is to perform rehabilitation exercises under the direction of a physiotherapist. The financial requirement to receive the treatment is not easy, so the family can suffer from financial burden. The resolution of this is a new virtual reality rehabilitation problem, which uses sensor tools to capture and recognize movements. The rehabilitation program requires physiological exercises like flexion, extension, abduction, adduction, enlargement, sleeves, dorsiflexion, plantar flexion, and rotation of various joints in patients with muscular and neurological trauma and disorder. In the existing literature studies, most of the researchers have focused on the detection of human activities like standing, sitting, sleeping, walking up and down stairs, etc., but very less attention was focused on the recognition and classification of rehabilitation physiotherapy exercises which is a multifaceted area of HAR.

Hitherto, HAR has been widely used in numerous applications, like gesture recognition gait analysis, human-computer interaction, home behavior analysis, personal health system, video surveillance, and antiterrorism monitoring [9-16]. It has the ability to learn in advance from raw data around human activities. Currently, HAR is a popular research track, due to progression in the field of human-computer interaction. Generally, there are two types of HAR: sensor-based and video-based. Sensor-based HAR depends on the data learned through keen sensors. Due to the development of ubiquitous computing and sensor technology, sensor-based HAR is more frequently used. To improve recognition accuracy, researchers have developed various types of sensing technologies such as techniques based on static and dynamic sensors. The video-based HAR takes advantage of the data acquired through various kinds of cameras to determine human activities [17], which is becoming popular due to the reduced complexity and ease of availability of different kinds of cameras. In this research article, for the detection of rehabilitation physiotherapy exercise, the dataset is collected through an RGB camera, and then a 3-Layer CNN-LSTM algorithm is applied for the detection of rehabilitation physiotherapy exercise. The 3-Layer CNN-LSTM algorithm seeks to leverage the power of merging both CNN and LSTM and address the deficiencies of existing approaches, laterally with the following characteristics: (1) the model is robust enough to perform equally well or better on input data, (2) it is evaluated on our self-created complex dataset, having rehabilitation physiotherapy exercises, (3) extracting and classifying activity features automatically, (5) and having better or at least same accuracy as of the existing DL approaches laterally with fast convergence speed and good generalization ability.

The manuscript is organized as follows: in Section 2 we take a look at some current techniques for HAR which is using machine learning and deep learning approaches. Section 3 explains CNN and LSTM algorithms and data preprocessing for the proposed model. Moreover, it contains a detailed overview of the 3-Layer CNN-LSTM model and its implementation. In Section 4, the performance of the 3-Layer CNN-LSTM model is explained along with their experimental results. Section 5, concludes the research work with a brief summary.

2. Literature Review

Machine learning models are used to learn the fundamental connections in data through experience while performing some tasks and making decisions without explicit instructions [18]. For a very long time, ML models have been used widely for HAR. Different types of models which can apply for HAR depend on data type, the volume of data, number of activities, similarities among activities, and number of activity classes. The existing ML models such as hidden Markov model (HMM) [19], linear discriminant analysis (LDA) [20], random forest (RnF) [21], logistic regression [22], support vector machine (SVM) [23], decision tree (DT) [24], histogram oriented gradient (HOG) [25], and K-nearest neighbour (KNN) [26] are used for human activity classification. Nevertheless, for precision and accuracies of the abovementioned algorithms, the selection of different parameters like the method of distance calculation and the number of neighbors for KNN, choice of the kernel for SVM, the tolerance value for LDA, and the number of trees for RnF plays a significant role which should be considered carefully [27-29]. These algorithms have achieved remarkable classification accuracies, nonetheless, it requires a lot of hand-tuning to formulate the data, feature engineering, preprocessing, and domain knowledge amongst others. These methods are not suitable in scenarios like indoor environments where confidentiality is required. Some of the other approaches are highly vulnerable to illumination disparities and background changes which are restraining their practical use.

A unique biometric system for detecting human actions in 3D space is proposed in [30] in which joint skeleton angles recorded through an RGB depth sensor are used as input features. The angle information is stored using the sliding kernel method. Before lowering the data dimension, the Haar wavelet transform (HWT) is used to maintain feature information. For dimensionality reduction, an averaging technique is applied to reduce computing costs and result in faster convergence. Moreover, the system may be used for elderly care and video surveillance, but there are a few drawbacks to the suggested approach. First, improper skeletal detection leads to inaccurate angle calculations, which causes the classification to be automatically misled. As the system is trained on activities in two directions at various angles and positions, there may be some confusion when attempting to recognize an activity due to probable similarities in the positions and angles of different activities. RGB-D images are beneficial for action recognition; however, the computational complexity of the learning model grows rapidly as the number of frames grows. As a result, the system becomes more complex and slower. Thus, instead of an RGB-D camera, a simple RGB camera could be explored to broaden the applications of the HAR system. Activity sequence recording, function extraction, model implementation, and finally identification are the four essential steps in vision-based HAR [31, 32]. The Kinect-based rehabilitation training system has arisen recently, which is utilized in most research studies. Some researchers focus on skeleton data while most of the researchers use RGB-D data, which is due to the fact that Kinect is based on depth sensors and uses structured light, which is not accurate. On the basis of depth data, it obtains the skeleton data, which is of low quality and high noise. Thus, the data obtained through skeleton joints is not accurate and has outliers which are declining the model performance.

Researchers have turned towards DL techniques for the detection of complex human activities. DL techniques extract features automatically from raw data during the training phase and have produced remarkable results in many activity recognition tasks. It has tremendous applications in the field of HAR, as the process of extracting features and classification are performed simultaneously. The RNN-LSTM approach used in [33, 34] has achieved outstanding performances and shown excellent results when compared to traditional hand-crafted practices, nevertheless, it exaggerates the temporal and understates the spatial information of input data. In many tasks such as NLP and speech recognition amongst others, CNN achieved better or at least similar performance to that of recurrent neural network (RNN). Due to this trend, recently CNN is widely used in the literature to grab the HAR problems. Numerous studies showed that CNN-based approaches are far better than traditional hand-crafted approaches since CNN has the ability to learn complex motion features [35]. The presented DL models for HAR are having simple architecture and great accuracy. However, these models were tested on simple datasets like standing, sitting, sleeping, walking upstairs and downstairs, and do not have a good generalization ability.

The main goal of this research work is to design a DL-based algorithm for automated detection of rehabilitation physiotherapy exercises. In the suggested 3-Layer CNN-LSTM model, data is fed to the convolutional layer for the extraction of useful features, and for classification, it is passed to LSTM to recognize the rehabilitation exercise. Batch normalization (BN) is applied to stabilize the learning process and reduce internal covariate shift. The model automatically detects the set of rehabilitation physiotherapy exercises and classify them under certain categories.

3. Methodology

For rehabilitation of stroke patients mostly two conventional techniques are used: artificially and robot-assisted. However, due to the cost of intelligent robots and their supplementary maintenance, robot-assisted techniques are difficult to be used. Moreover, due to the shortage of healthcare providers, artificial form of rehabilitation is also hard to be accessed. As we know that rehabilitation of stroke patients is a long-lasting process due to which both robot-assisted and artificial forms of rehabilitation are not feasible. Therefore, an automated rehabilitation training method is needed to address this issue. Consequently, we designed an automated model to recognize the physiotherapy exercise while keeping in view the existing HAR approaches. The architecture of the proposed algorithm is shown in Figure 1.

Figure 1

Architecture of 3-Layer CNN-LSTM model.

The 3-Layer CNN-LSTM model seeks to leverage the power of merging both CNN and LSTM. The main approach is divided into two sections, the detection of physiotherapy exercise and its classification. The first section contains data collection, preprocessing of images, dimensionality reduction, and data augmentation. The second section is using a combination of DL models to evaluate the features and classify the physiotherapy exercises efficiently. To test the model's efficiency, various experiments were performed for the detection and classification of physiotherapy exercises. The main components of the algorithm are discussed briefly below.

3.1. Convolutional Neural Network

It is a type of deep neural network (DNN) having multiple layers. Its concept is originated from the receptive field and neural cognitive machine which is more sophisticated than a traditional neural network. In the presence of additional deep layers, the DNN model learns deeply compared to other shallow neural networks. Dealing with the problem of image classification and recognition, the CNN has high distortion tolerance due to its spatial structure and weight sharing mechanism [36]. The fundamental structures of the CNN are the amalgamation of weight sharing, subsampling, local receptive field, and dimensionality reduction during feature extraction. To reduce the complexity of the model, enhance their performance and efficiently regulate the number of weights, the weight sharing mechanism is used. CNN map input image data to an output variable i.e., it takes the input image data, processes the data, and predict the image class efficiently. Input data is in the form of a two-dimensional vector and CNN deals with it in a better way. In 3-layer CNN-LSTM model, we used CNN to extract useful features from the input image matrix. In this step, the physiotherapy exercise image is taken as input which is processed to extract features from it. In this work, the LSTM is used to classify the input data under certain categories. The process of CNN for feature extraction is described in detail in Section 3.3. The learning and classification technique of CNN is described mathematically through (1). In equation (1), Z is the set of inputs, W is the set of weights, and B is the bias operation.

3.2. Long Short-Term Memory

LSTM is used to evade the problem of gradient vanishing or gradient exploding during training. The back propagation (BP) algorithm is used to update the weights of the neural network. The BP algorithm first calculates the gradient using the chain rule and then updates the weights of the network on the base of the calculated loss. The BP starts from the output layer and the whole network is traversed towards the input layer which is facing vanishing gradient or exploding gradient problems while updating the weights in DNN. So, to avoid the said problem of gradient explosion or gradient vanishing during training traditional RNNs, an LSTM algorithm is proposed. Furthermore, RNNs are unable to memorize long sequences of data, while LSTM efficiently deals with it. LSTM is a type of RNN and the building block of artificial neural network which is having additional memory cells for time steps and remembers the past information. The process diagram of LSTM is shown in Figure 2. It is capable to remember and learn long-term sequences. LSTM consists of 4 different components: input gate (I), output gate (O), forget gate (F), and cell state (C) at time step (t) [37]. The past information is stored in the state vector of C. The I decides how to update the state vector using the current input information. The data which are added to the state from the current input is represented through L vector. Z represents input vector at time step t, H, and H is the current and previous cell output, C and C is the current and previous memory cell, (x) is elementwise multiplication, and W, U represent weights of the four gates i.e. I, O, F, and C. Due to this structure of LSTM, it is applied to learn efficiently complex sequences of data.

Figure 2

Process diagram of LSTM.

The σ and Tanh is a nonlinear activation function and UI, WI, UF, WF, UO, WO, UL, and WL are the respective weights which are having M × 2N dimensions, where M shows the number of memory cells and N shows the dimension of the input vector. The mathematics of LSTM behind the whole process is formalized in [38] as shown in equations (2) to (7).

3.3. Data Representation and Feature Extraction

Data representation is the first section of the suggested approach which contains data collection, preprocessing, and feature extraction. Data for this research study were collected through an RGB camera, from participants performing exercises under the direction of a physiotherapist. Data augmentation is used to reduce overfitting and enlarge the dataset artificially as shown in Figure 3. After the collection of raw images, the next step is to preprocess them prior to the implementation of any proceeding functionalities. Data preprocessing involves data cleaning such as noise removal, resizing and filling, or removing null values.

Figure 3

Data augmentation.

Data for each rehabilitation physiotherapy exercise are combined in one file named as “Categories” and preprocessed by resizing each image to reduce complexity. This whole process is shown in Algorithm 1.

Algorithm 1

Data preprocessing for training.

To minimize the set of features and enhance the efficiency of the classification algorithm, useful features are extracted from the data. The extraction of correct features is an exciting job for the recognition of physiotherapy exercises for which CNN is used. In feature extraction phase convolution operation, pooling operation and ReLU (rectified linear unit) activation function is applied as shown in equations (8) and (9), respectively

The physiotherapy exercise data named as, New_array is taken as input and passed through convolutional layer followed by ReLU, pooling, and dropout layer to extract useful features from it. The input is of order 2 matrix with HxW (rows, columns), H and W are indexed as (m, n), where 0 ≤ m ≤ H, 0 ≤ n ≤ W. The final and useful feature map values i.e., F_y is obtained through convolutional operation layer 3

The activation function is applied at every layer to make the model capable of solving nonlinear problems as shown in equation (9), while to minimize the computational load, max pooling and dropout technique is used.

3.4. Classification

The 3-layer CNN-LSTM is used as a learning algorithm for feature extraction and its classification. The whole process of feature extraction and its classification under certain categories are shown in Algorithm 2. For classification of rehabilitation physiotherapy exercise pretrained LSTM is applied as explained in detail in Section 3.2. The pretrained LSTM is followed by fully connected (FC) layers, batch normalization (BN) layer, and SoftMax function. The proposed algorithm makes it possible to see that during the training phase, the accuracy is much more than the testing phase, which is because of overfitting and outcome a complication in the regularization and balancing of hyperparameters.

Algorithm 2

3-layer CNN-LSTM model for the detection of rehabilitation exercise.

BN is applied to stabilize the learning process and reduce internal covariate shift. During the training period, at each middle layer, the BN calculates the mean and variance as shown through equations (10) and (11). For each layer, the normalized input is gained from the previously calculated mean and variance as shown in equation (12).

During the training of the network, γ (standard deviation) and β (mean parameter) along other parameters are learned. The final mean and variance equations for testing the model are shown in equation (13) to (15), respectively [39]. In these equations' “j” shows the number of batches where each batch has “m” samples. The final mean and variance are assessed from the previous mean, variance formulas calculated for each batch during training. Consequently, after all in BN, the layers are normalized through final mean and variance procedures.

The purpose of training the model is to adjust the filter weights such that the predicted class should be as close as possible to the actual class. During training, the network runs in the forward direction to get the resultant predicted value. The loss function is calculated to evaluate how well our proposed model is working. To compare the predicted value with the corresponding target value through continuous forward pass running, the total loss at the last layer is obtained. The loss is guiding the model to update parameters to reduce the error rate. The relative probability of real values at the output layer is calculated through the SoftMax function to recognize the rehabilitation exercise. A short comparison of the proposed model to the current state-of-the-art models is given in Table 1.

Table 1

Comparison of 3-layer CNN-LSTM model with other standard models.

S. No	Other standard models	Proposed CNN-LSTM model
1	RNN-LSTM model used in [33] for HAR system and achieved great accuracy, but this model exaggerates the temporal and understates the spatial information as both of the models best fit for temporal data.	CNN is used for feature extraction and selection of useful features, while LSTM is used for exercise recognition. This model maintains a balance between spatial and temporal information.
2	In [40], LSTM-CNN model is used for activity recognition. The LSTM is used before CNN to process input data which is not efficient for the processing of spatial input data.	In the proposed model, 3-layer CNN is applied first to process spatial input data. The data is then fed to the LSTM layer to further refine the extracted data and detect the rehabilitation exercise.
3	The CNN model for exercise recognition was tested and observed that CNN learn too many complex parameters of about 2,575,753 during training.	The model learned about 392,765 parameters which conclude that CNN-LSTM model is lightweight which has reduced complexity and achieved better accuracy.
4	In [30], KNN is applied to recognize human activity which fails to address occlusion, deformation, and viewpoint variation, as KNN is using hand-crafted techniques.	The 3-layer CNN-LSTM model learns activity feature automatically and handles these issues efficiently. We used an RGB camera instead of Kinect sensors to reduce complexity and processing time.

4. Results and Discussion

The model is trained in a fully supervised manner, and the gradient is backpropagated from the SoftMax to CNN layer to reduce the loss. Through randomly selected values the bias and weights are initialized at each layer.

4.1. Hyperparameter Selection

During classification, the model performance is greatly affected by the selection of hyperparameters. The impact of different hyperparameters such as the number of convolution filters, batch size, learning rate, kernel size, pooling size, epochs, and type of optimizer is observed on model performance and explained as follows.

To increase the number of convolution filters, the model learns more complex features which ultimately increases the number of parameters and causes overfitting issues. So, the accurate and balanced selection of filters at each layer is important. At the first layer, we used 64 filters. In the second convolutional layer, the number of filters is doubled compared to the first convolution layer and so on, to cope with downsampling caused by the pooling layer. The selected number of filters shown in Table 2 outperforms the other combination of filters at different layers. In the start, a reduced size filter is used to learn low-level features, nevertheless, for high level and specific features, large size filters perform better. In layer 1, kernel size of (5,5) is selected, whereas in layers 2 and 3, kernel size is reduced. The idea behind selecting a large filter size at the start is that it read generic features in one value and its effect is more globally on the whole image, but missing local features. In layer 2 and layer 3, a small filter size is used to learn local and specific features. The number of filters and size of the filter are selected by the hit and trial method.

Table 2

List of hyperparameters selected for training.

Processing stage	Hyperparameters	Values selected
Convolution_1	Filters	64
	Kernel size	5
	Stride	1
	Max pooling	4
Convolution_2	Filters	128
	Kernel size	3
	Max pooling	2
Convolution_3	Filters	256
	Kernel size	3
	Max pooling	2
Training parameters	Learning rate	0.1
	Epochs	36
	Batch size	32
	Optimizer	Adam

We used different batch sizes and monitor the model performance. By selecting 32 batch sizes, the highest accuracy is achieved. An optimal learning rate of 0.1 along with 36 epochs is used in the training stage to improve the fitting ability of the model. The impact of changing learning rate and the number of epochs was studied and it was concluded that by reducing the learning rate, the process takes a long time to converge while the high learning rate results in the process to converge quickly. It is observed that the learning rate and the number of epochs have an inter-relationship with each other and affect model performance. During training the 3-layer CNN-LSTM model, Adam optimizer is used which has the best fitting effect on model performance and gives the highest accuracy. For training purposes, numerous combinations of hyperparameters are used and tested by using the hit and trial method for parameters selection, and finally, the best parameters giving the highest performance results are selected. The list of selected hyperparameters is shown in Table 2.

4.2. Dataset of Rehabilitation Exercises

A physiotherapist employs many complex exercises for the rehabilitation of stroke patients depending on the type and severity of stroke. Rehabilitation exercises include but are not limited to flexion and extension of the neck, flexion and extension of the trunk, flexion and extension of the knee joint, flexion and extension of the hip joint, flexion and extension of the wrist, abduction and adduction of the upper limb, and dorsiflexion and plantar flexion of the foot as shown in Figure 4.

Figure 4

List of rehabilitation exercises, from left to right: (a) extension of the neck; (b) rotation of neck; (c) flexion of the trunk side view; (d) flexion of the trunk front view; (e), (f) extension of the elbow joint (front and side view); (g) dorsiflexion of the foot; (h) plantar flexion of the foot; (i), (j) extension and flexion of the knee joint; (k) extension of the trunk; and (l) flexion of the wrist.

In this research work, we generated our own dataset under the direction of a physiotherapist consisting of 2250 different samples. The data is recorded from participants performing different rehabilitation exercises through an RGB camera. The participants include males, females, and children having up to 40 years of age. The description of the dataset is given in Table 3.

Table 3

Description of the dataset.

Total samples of rehabilitation exercise: 2250
Number of participants: 20
Rehabilitation exercise	No. of samples
Flexion and extension of neck	332
Flexion and extension of trunk	328
Flexion of knee joint	348
Flexion and extension of wrist	489
Dorsiflexion and plantar flexion	419
Abduction of upper limb	334

4.3. Training

The model training is performed on a Dell laptop with an Intel Core i7 processor and 16 GB RAM equipped with 64 bits operating system The classification model is implemented in Python 2.7.0 with Jupyter notebook. The main theme of the training is to adjust the filter weights such that the predicted class should be as close as possible to the actual class. The dataset is divided into two parts. The first part is having 80% of data which is used for training purposes while the second part is having 20% of data and is used for testing the efficiency of the model. The Adam optimization algorithm is used for adjusting the weights in such a way to move from a large loss point to a small loss point using the error backpropagation method for optimization. According to the activation function, the weights are then updated.

4.4. Performance Evaluation Metrics

To scrutinize the performance of the model various evaluation metrics are used, which shows the reliability of the model in examining the rehabilitation exercise. The most common metrics used for performance evaluation are recall, f1-score, precision, and accuracy [41-44].

4.4.1. Precision

It is the ratio of true positive (TP) to TP and false positive (FP) observation, which is predicted positive and is calculated as

4.4.2. Recall

The recall is the ratio of the predicted true positive observation to true positive and false negative (FN) observation, which is actually positive and is calculated as follows:

4.4.3. F1-Score

F1-score is the weighted average of recall and precision and calculated as

4.4.4. Accuracy

It is the ratio of correctly classified activities to the total number of classified activities.

4.5. Results

The confusion matrix of rehabilitation exercises is shown in Figure 5, which shows the true label at the y-axis and the predicted label at the x-axis. The numbering from 0 to 6 shows the set of different exercises e.g. dorsiflexion, neck exercise, plantar flexion, trunk extension, trunk flexion, wrist extension, and wrist flexion, respectively. The classification report of 3-layer CNN-LSTM along with the CNN model is shown in Table 4.

Figure 5

Confusion matrix of 3-layer CNN-LSTM model.

Table 4

Classification report of 3-layer CNN-LSTM and CNN model.

	CNN-LSTM model			CNN model
Exercises	Precision	Recall	F1-score	Precision	Recall	F1-score
Dorsiflexion	0.95	0.96	0.95	0.93	0.99	0.96
Neck exercise	0.87	0.88	0.88	0.87	0.87	0.87
Plantar flexion	0.94	0.87	0.91	0.91	0.86	0.89
Trunk extension	0.94	0.87	0.90	0.93	0.81	0.86
Trunk flexion	0.89	0.93	0.91	0.95	0.84	0.89
Wrist extension	0.91	0.92	0.92	0.85	0.90	0.88
Wrist flexion	0.84	0.90	0.87	0.84	0.90	0.87

The performance evaluation of the 3-layer CNN-LSTM model is obtained in terms of precision, recall, f1-score, and accuracy which is calculated according to equations (16) to (19). The performance of the model for discrete rehabilitation exercises is evaluated through Figure 6. To validate the dominance of the suggested 3-layer CNN-LSTM model, it is compared with KNN [30] and CNN models. The overall accuracy achieved by KNN, CNN, and 3-layer CNN-LSTM model is given in Table 5 while represented graphically in Figure 7. Consequently, we see from Table 5, a gradual decrease in test errors and an increase in accuracy. The model achieved the highest recall of 96%, precision of 95%, f1-score of 95% for dorsiflexion of the foot, and lowest recall of 90%, precision of 84%, and f1-score of 87% for the wrist flexion as shown in Figure 6.

Figure 6

Performance in terms of precision, recall, and f1-score.

Table 5

Performance comparison of different models.

S. No	Model	Accuracy (%)
1	KNN [30]	86.1
2	CNN	89.9
3	3-layer CNN-LSTM	91.3

Figure 7

Line graph showing the average accuracy of CNN-LSTM and CNN model.

The precision, recall, and f1-scores are calculated to validate the performance, in case the dataset in a class is imbalanced and accuracy may produce deceptive results.

In 3-layer CNN-LSTM model, the LSTM which is a variant of RNN is the primary learning element and produced better or at least the same accuracy compared to other state-of-the-art models on the prescribed datasets. The model was tested on different types of data, which the model had not seen before, and observed that the CNN-LSTM model has the same best accuracy. It confirms that the model is not overfitted and is performing better in situations like that of occlusion, viewpoint variation, and deformation.

5. Conclusion and Future Work

Deep learning models have powerful learning abilities in dealing with deformation, viewpoint variation, occlusion, and background switches. In the suggested model, a DNN algorithm is implemented that combines CNN and LSTM as 3-layer CNN-LSTM for the detection of rehabilitation exercises. After the fully connected layer, a BN layer is added to reduce the internal covariate shift and speed up the convergence procedure of the model. In the proposed architecture, the data collected by an RGB camera under the direction of a physiotherapist is fed into a 3-layer CNN followed by an LSTM layer. The CNN along with LSTM is making the model proficient in learning the spatial and temporal dynamics at various time slots. The parameters are learned over CNNs and further classified by LSTM to attain better accuracy and preserve a high recognition rate. In the future, the model shall be trained on more complex datasets to detect complex rehabilitation physiotherapy exercises. Moreover, the algorithm will be updated in such a way that can be used at home for a patient to carry out the prescribed rehabilitation exercises without direct in-person supervision of physical therapists.