1D Multi-Point Local Ternary Pattern: A Novel Feature Extraction Method for Analyzing Cognitive Engagement of students in Flipped Learning Pedagogy.

Flipped learning is a blended learning method based on academic engagement of students online (outside class) and offline (inside class). In this learning pedagogy, students receive lesson any time from lecture videos pre-loaded on digital platform at their convenience places and it is followed by in-classroom activities such as doubt clearing, problem solving, etc. However, students are constantly exposed to high levels of distraction in this age of the Internet. Therefore, it is hard for an instructor to know whether a student has paid attention while watching pre-loaded lecture video. In order to analyze attention level of individual students, captured brain signal or electroencephalogram (EEG) of students can be utilized. In this study, we utilize a popular feature extraction technique called Local Binary Pattern (LBP) and improvise it to develop an enhanced feature selection method. The adapted feature selection method termed as 1D Multi-Point Local Ternary Pattern (1D MP-LTP) is used to extract unique features from collected electroencephalogram (EEG) signals. Standard classification techniques are exploited to classify the attention level of students. Experiments are conducted with the data captured at Intelligent Data Analysis Lab, NIT Rourkela, to show effectiveness of the proposed feature extraction technique. The proposed 1D Multi-Point Local Ternary Pattern (1D MP-LTP)-based classification techniques outperform traditional and state-of-the-art classification techniques using LBP. This research can be helpful for instructors to identify students who need special care for improving their learning ability. Researchers in educational technology can extend this work by adopting this methodology in other online teaching pedagogy such as Massive Open Online Courses (MOOC).

Introduction

Knowledge is the foundation upon which civilizations are built. Learning is the single most primary attribute required for the evolution of human beings. Over the years, the purpose of learning has remained the same, but methods have evolved. In ancient India, the Gurukul system was prevalent, with students staying in ashrams of their gurus and learning under the shade of banyan trees. Modernization and urbanization led to concrete buildings to be used as schools and institutions to impart education. Teachers deliver lessons inside a classroom through verbal instruction using boards or multimedia projectors while students take notes, listen and understand the lessons. Doubt clearing and assessment happen inside the four walls of the classroom. This is the traditional classroom pedagogy, which is still ubiquitous and relevant today. This provides an opportunity of physical interaction between the teachers and the students. The teacher can address the doubts and difficulties of each student in-person. However, this face-to-face mode of learning has a few drawbacks. A greater emphasis is given in delivering the lectures and completing the syllabus than practice or problem solving, due to time constraints. Thus, students lack the capacity to solve the practical problems using the theoretical knowledge. Due to the larger batch size of around students, it becomes nearly impossible to identify the students who are inattentive. However, this is not the only method of learning today. The twenty-first century world is synonymous with digital revolution and Artificial Intelligence (AI). AI has spread its roots into almost every domain today. In the past few years, there has been a massive infusion of technology in the education domain. Innovative learning techniques like Massive Open Online Courses (MOOC), e-learning, m-learning (mobile learning) have enabled seamless access to learning for students anywhere anytime. Modern education tools like projectors, smart-boards, audio-video devices, digital textbooks, even seating arrangements [1], all powered by AI have provided useful alternatives for learning and can even improve the traditional approach. The shortcomings of the traditional classroom can be addressed by online learning and m-learning, where live online lectures or pre-recorded videos are available for the students at their fingertips. The instructors can also reap benefits of online assessment methods [2-4]. However, online learning comes with its own share of shortcomings. There is no scope for direct physical interaction between students and teachers. Problems such as lack of focus on critical thinking and problem solving still prevail in this mode. The instructor cannot identify the inattentive students, which hinders the potential success of online learning.

The blending of traditional and online learning methods can be a solution to this issue. One such popular learning mode is Flipped Learning [5]. In this mode, students complete their lessons in their homes or hostels. They can access reading material and pre-recorded lecture videos outside class as per their convenience. Inside the classrooms, the students engage in discussions and get their doubts and queries resolved by the instructor. This gives the instructor sufficient time to focus on imparting critical thinking skills and real-life problem solving inside the classroom. Unfortunately, the flipped classroom is vulnerable to shortcomings as well. It is hard for an instructor to know whether a student paid attention while watching pre-loaded lecture video. Monitoring the attention engagement levels of learners becomes important in this context. Sensors can be used to monitor their activities as they watch video lectures. The signals recorded by these sensors can be then analyzed and used to classify their attention levels [6, 7]. More signals can be captured using multiple sensors on different body parts [8]. But this multi-model approach needs expensive equipments. In addition, the students may not be able to give natural attention and get irritated when more than one sensor are put on their bodies. Multiple sensors may produce more noise compared to using single sensor. Coordinating multiple sensors is another major concern and system becomes complex. Therefore, researchers focus on using only single sensor in various domains [1, 9–11].

In this paper, we propose a technique to record EEG signals of students while they watch video lectures, extract features from the recordings, and use them to classify students’ attention levels. We call our proposed method 1D Multi-Point Local Ternary Pattern (1D MP-LTP). This method involves the use of Discrete Wavelet Packet Transform (WPT) to extract the useful features and reduce the dimension of data. Our proposed 1D MP-LTP method is then applied over it to obtain the desired features. Ternary Patterns are designed on varying block sizes for performing comparative analysis. The obtained Ternary Patterns are also converted into Binary Patterns in order to analyze and understand the need and effectiveness of Ternary patterns. We obtain several feature sets from the different conversion operations on different block sizes. To validate these features, standard classifier models and evaluation metrics are used. The classifiers using our proposed 1D MP-LTP method outperform traditional and LBP-based classifiers. EEG signal is a key attribute for analyzing the cognitive state of the student and has been used in different sub-domains of educational technology [1, 12, 13]. However, its utility is yet to be explored in the field of flipped learning. The novelties and contributions are summarized as follows.This work is an extended part of our accepted paper in proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT 2021) [14].

We propose the 1D MP-LTP method for novel feature extraction from brain wave signals in order to improve the learning ability of the student in the flipped learning pedagogy.

Data acquired at our lab for research on flipped learning has been utilized. Experimental results of our two proposed methods are evaluated with standard performance metrics such as accuracy, recall, precision, and F1 for classification purposes. Results showed the effectiveness of our proposed approach in the field of flipped learning pedagogy.

This paper is organized as follows. We provide a brief literature related to this article in “Related Research”. A background of the proposed method is laid in “Background”. In “Dataset Description” and “Proposed Method ”, dataset description and the proposed method have been described, respectively. Experimental results are demonstrated in “Experimental Results ”. Finally, “Conclusions and Future Scope ” concludes this work and discusses possible directions and scope of this work in the future.

Related Research

In this section, we describe related research work about academic engagement of students in Flipped Learning (FL) and feature extraction using Local Binary Patterns (LBP) [15] in various domains such as image processing [16, 17], signal analysis [18, 19], healthcare [20], emotion recognition [21], bio-metric identification [22], rotary machines [23], etc.

Student Academic Engagement in the Flipped Learning

Extensive research has been accomplished to enhance the quality of education in traditional as well as online and blended learning. Cognitive Analysis in Education (CAE) involves understanding the underlying mechanism of thought process while learning. Rogaten et al. [24] review a rich but diverse variety of adopted methodologies to measure behavioral and cognitive learning gains.

To understand how efficiently a person is understanding or learning the lectures, cognitive engagement needs to be assessed. It becomes a challenging task to differentiate between cognitively engaged students from the rest. This is because, the unengaged student usually imitates the engaged ones for the sake of evading from the hurdles of learning. Giacomone et al. [25] take up a research gap in mathematics education research to provide an exhaustive characterization of mathematical activity carried out by students. The author uses the answers of 30 prospective primary education teachers to a specific mathematics problem on fractions using area and tree diagrams. Roohr et al. [26] examine learning gain of college students based on their performances in mathematics, writing, reading, and critical thinking as assessed by the ETS Proficiency Profile (EPP). They concluded that the student gained less knowledge in the first two years (first and second year), compared with last two years (third and fourth year). While comparing with the conventional curriculum-based learning, R. Amanda et al. [27] examine the effectiveness of Team-Based Learning (TBL) on persistent retention of knowledge. An elaborate analysis is carried out in [28], which tries to show the effect of lecture video along with in-video assignments on students. In a recent study, [29] by Pappas et al., extensive research has been shown to analyze student engagement in adaptive learning through feedback, monitoring, and personalizing of contents. Questionnaire data, electrodermal signals, EEG signals, Click-streams, and eye-tracking were used to measure arousal, cognitive load, response time, attention, and emotions for analysis. Sharma et al. [30] also utilize wristband data, eye-tracking, facial video, and EEG data as features to predict performance levels of students in a “grey-box” approach on building pipelines for educational data. Mangaroska et al. [31] explore the utility of multiple e-learning systems to enhance learning methods using cross-platform analytics. All these works are directly related to the development of Flipped Learning models as a pertinent source of learning.

Subramaniam and Muniandy [32] show the comparison of performances in flipped learning (43 students) and traditional settings (41 students) based on emotional, behavioral, agentic and cognitive engagement levels. Sojayapan and Khlaisang [33] divided students into groups for flipped learning, traditional learning and online classroom to investigate their team learning ability. From t test analysis of post-test and pre-test results, flipped classroom was concluded to be more effective. To enhance learning and motivation in MOOCs, eye movements, gaze movements, stimuli and reading patterns are analyzed in [34].

Giannakos et al. attempted to capture psychological data from devices that could be worn on wrists [35]. The authors used feature extraction techniques such as FFT, statistical measures. They were able to prove successfully that psychological senses could be a plausible replacement to measure attention and focus levels of students. Similar to attention, relaxation state of a student can also be analyzed [36]. The authors compared the mobile polling and Just-in-Time Teaching (JiTT) strategy. In [37], the authors compared traditional click-stream models to current models based on psychological responses. Along with EEG data, eye-tracker, wristbands and webcam were used to collect psychological data. The findings showed EEG data and eye-tracking mechanisms to be especially dependable for attention measurement. A SVM-GA methodology has been adopted by Chen et al. [9] to analyze the attentiveness of the students using EEG data. The experiment employs Discrete Wavelet Transform and is used for feature extraction. A novel approach involving Genetic Algorithm (GA) is used to select seven optimum features and further pass through the SVM (Support-Vector Machine) model to classify the students with high or low attention rates. The model also identifies the portions of the video lectures on which students pay less attention. Similarly, sustained cognitive load, emotion, attention, and learning performance for different kinds of video lectures have been evaluated by authors in [38-41]. Chen and Wu [39] used three different kinds of videos described as voice-over presentation type, lecture capture format, and picture-in-picture method. Along with evaluating attention, they also evaluated stress levels.

Feature Extraction by Local Binary Patterns

In the fields of image processing [42-44], signal analysis [21, 22], healthcare [20], emotion recognition [21], bio-metric identification [22], etc., local information is also necessary while extracting notable features. This led to the foundation of a concept called Local Binary Patterns. The idea is to utilize the intensities of neighboring points (pixels in the case of image) and develop a representative value for that point. Thus the entire feature vector is transformed into a new one, which has the texture significance directly associated with it. This updated feature vector is then used for classification purposes.

Local Binary Pattern has found immense utility in the field of medical diagnosis. The LBP has been utilized to detect epileptic seizures in the brain from EEG signals [20, 45]. Epilepsy diagnosis is also the target in [18], where Discrete Wavelet Transform has been utilized to decompose EEG signals into 5 levels using 1-Dimensional LBP. Quad Binary Patterns are also used to diagnose epileptic [46]. One-dimensional LBP has been used to detect Alzheimer’s Disease [47]. A very recent work in the field of medicine where a Multi-Kernel Local Binary Pattern has been used to classify COVID-19 pneumonia from medical images [48]. LBP is extended to classify emotions from EEG signals. The extended LBP named as Fractal Patterns uses edges of the image as central point unlike traditional in LBP [21]. LBP is also extended to one-dimensional Local Difference Patterns for classification ECG signal [22]. LBP has been extended to 1-D Local Ternary Pattern (LTP) on rotary machines [23]. While two-dimensional LBP has been used in the domain of image processing, the same can be extended to signals with one-dimensional LBP or LTP [23, 47].

As we discussed in “Student Academic Engagement in the Flipped Learning”, EEG signals, electrodermal signals, eye gaze, gaze gesture or hand movement are analyzed to know the degree of engagement of the students with learning contents such as lecture video [2–4, 13, 49–55]. It is found that EEG signals are the most deciding factor for cognitive ability. However, EEG signals need to be processed to extract important features for further analysis [56, 57].

On the other hand, LBP (“Feature Extraction by Local Binary Patterns”) has been found as an effective feature extraction technique in many domains such as image processing, signal analysis, healthcare, emotion recognition, bio-metric identification, rotary machines, etc. This motivates us to explore LBP as feature extraction technique to analyze EEG signals obtained from one-channel headset which is easy to wear and cost-effective. In the current scenario (COVD 19) online learning is prevalent; however, teachers may not be able to know the engagement level of student. This will help teachers analyze the engagement level of student.

Background

In our proposed method, we exploit two important feature extraction techniques, namely Local Binary Pattern (LBP), and Discrete Wavelet Packet Transform (DPT), which are discussed next.

Local Binary Pattern (LBP)

In the fields of image processing or signal analysis, local information can be useful while extracting notable features. Ojala et al. [15] introduced a concept for the comparative study of texture measures in 1996, known as local binary pattern (LBP). LBP is mainly focused on the texture features of the image. Usually, a block of size is chosen. And the central pixel is compared with all of its neighboring pixels. Each of the neighboring pixels are assigned values based on their comparison with the central pixel. If a neighbor pixel value is less than the value of central pixel, it is assigned 0, else 1. After assigning the binary values, the binary string is generated by going over all the neighbors (clockwise or anticlockwise). The binary string is then converted to its decimal value, which is the updated value of the pixel point. A histogram is then generated based on the frequency of each number occurring. This histogram is used as the feature vector. An example of the process is shown in Fig. 1. This process is continued for all pixels. Several LBP [47, 58, 59] operators can be defined according to their neighbors as shown in Fig. 2.

Fig. 1

Local Binary Operator

Fig. 2

LBP operators with varying radii and number of neighbor points

Discrete Wavelet Packet Transform (DPT)

Wavelet Transforms are multi-resolution-based signal analysis techniques having their credibility in frequency as well as time domain analysis. Although Fourier transform has been historically used for frequency band analysis, Wavelet Transforms are more advantageous than the former. Fourier transform fails to provide information with respect to time, whereas Wavelet Transforms are used to receive required information in both frequency and time domains. In wavelet transform, the signal can be represented as the superposition of basic functions called wavelets. These wavelet functions are responsible in capturing information in both frequency and time domain.

The classical Discrete Wavelet transform (DWT) decomposes a signal into multiple orthogonal wavelets. The mother signal subjected to Discrete Wavelet transform provides two sets of coefficients symbolizing time-frequency domain representations. The low-ranged frequency coefficients termed as approximation coefficients are obtained using low-pass filters. Similarly, the high-ranged frequency coefficients are obtained using high-pass filters. The conventional DWT allows further decomposition of approximation coefficients and facilitates multi-resolution analysis. Tackling this very limitation, Discrete Packet Transform (DPT) [56, 57] allows the further decomposition of both approximation and detailed coefficients. The multi-resolution-based decomposition yields a tree-like decomposition facilitating decomposition at each level. The number of levels of decomposition and the choice of mother wavelet is specific to the use case being dealt with. Discrete Packet Transform, being more flexible and having better control, is more suitable over the conventional DWT.

Dataset Description

Thirty-one male and thirteen female participants were selected for our experiment. All participants were between 17 and 20 years old and belong to departments of our institute from B. Tech course (first and second year). The selection procedure of participants for the experiment was compliant with the guidelines of the Institutional Review Boards (IRB) of NIT Rourkela. For our experiment, we used NeuroSky’s MindWave Mobile2 (Fig. 3) headset1 which was connected to the computer via adapter or Bluetooth. This is done to capture the EEG signals which are the responses by brain’s stimulus. Sampling rate of headset is 512 Hz. The stimuli are continuous and the produced time signals are converted into the frequency domain subsequently.

Fig. 3

EEG headset NeuroSky’s Mindwave Mobile 2

Ground Truth

We need to assign labels to the recorded data for classification. For this, two tests are conducted, before (pre-test) and after (post-test) a student watches a lecture video. We computed Pearson correlation between two tests marks for a student and EEG signals with positive correlation are considered in the experiment.

The pre-test marks achieved by a student are considered, x which determines the prior knowledge possessed by the pupil on the video’s topic. The post-test marks are considered to be y, which indicates how much he/she learnt from the video lesson. Let the mean value of the post-test marks of all students is a and the difference between mean values of post-test and pre-test marks is b. We assume that student is speculated to have sufficient knowledge about the topic if x is greater than or equal to of total marks. His (her) EEG analysis will not be effective in the primary purpose of the experiment. In a case where a student obtains a marks in the post-test, this reveals that she performed worse than the average mass even after watching the video lectures. In addition, if the improvement over pre-test marks of a student, referred as (ImpX) (), is less than b, this indicates that the student did not even improve in his (her) own performance, which clearly shows lack of focus during the lecture video. This explains that the concerned student did not comprehend the lecture and it is labeled as Inattentive. All other cases are labeled as Attentive. Mathematically, the labeling approach is determined by using Eq. 1.

Proposed Method

In this section, we utilize the concept of LBP in the proposed 1D Multi-point Local Ternary Pattern (1D MP-LTP) method for generating novel features from EEG signals.

During pre-processing, each EEG signal corresponding to a lecture video is subjected to Discrete Wavelet Packet Transform (WPT). The WPT is used in decomposition of the signal into set of wavelets, which helps to get relevant features in time-frequency domain [60]. Each level of DPT reduces the dimension of data by almost half. In our work, after experimentation with various levels of decomposition, four levels of decomposition are carried out, the approximation coefficients of the fourth decomposition are used (Fig. 4). Having applied WPT, we use 1D MP-LTP method which is described below.

Fig. 4

EEG Data analysis filtering through DWT

Let be the single dimensional data obtained after applying WPT on EEG signal corresponding to a video watched by a student. We group k consecutive values in L, say to form one block. After grouping, we get non-overlapping blocks in total. In a single block, we assign a ternary bit to each value. This assignment of ternary operator is based on a threshold value . The value of threshold decides the boundaries for ternary values. In our study, the value of is taken as half of the mean amplitude () value of each record L.

We calculate the LTP value of each block. We convert each value of a block into ternary bits as follows. The center element is referred to as and neighbors (other elements) are referred to as , where i indicates the index of the element. Ternary values are then assigned to create the Ternary string. If is greater than or equal to a threshold then we assign a to element. If is greater than or equal to a threshold , we assign it as . For all other cases, we assign 0. We get a ternary string X using Eq. 2.where is the ternary bit position. This process is carried out for each block.

Obtained ternary strings from each block can also be converted into binary string. Therefore, we can obtain two primary variants of proposed method based on the string conversion used. These variants are termed as 1D MP- where we directly read string in the ternary format and 1D MP-, if we convert the ternary strings to binary.

We can further obtain two variants of 1D MP- based on the way we read the ternary string of each block (clockwise, anticlockwise). These variants are 1D MP-, and 1D MP-. The entire procedure is mentioned in Algorithm 1.

After encoding a string block using Local Ternary Operator, we replace in the strings with either 0 or 1. We can read the strings of ternary bits from left (anti-clockwise (A)) or right (clockwise (C)) end. Each ternary string is converted in four different ways, which are described as follows.We finally encode string to it corresponding decimal representations using the formula given in Eq. 3. We repeat this process for the next block and so on. At the end, we combine all the decimal-coded string blocks to obtain the features for each 1D data.where m is the base (in our case, 2 for binary and 3 for ternary) and k represents the string’s block size. In our experiment, we consider the values of k as 5, 7, and 9, respectively. The exact value of k is decided by trial and error method. We use odd numbers for the value of k, so that we get an equal number of cells on the both sides of the central (median) element. Performance of the proposed method was not satisfactory beyond the value of k equal to 9. So, we restricted to use these values (5, 7 and 9) only. The entire 1D MP-LTP process is shown in Fig. 5. In this way, we have two feature sets generated from 1D MP- method and four different feature sets generated from 1D MP- method. The entire procedure is mentioned in Algorithm 2.

Fig. 5

1D Multi-Point Local Ternary Pattern (1D MP-LTP)

1D MP-: Each block is read in anti-clockwise, and value -1 is replacement with 1. It generates lower binary codes. Lower binary code value must be greater than or equal to upper binary code.

1D MP-: Each block is read in clockwise, and value -1 is replacement with 1. It also generates lower binary codes.

1D MP-: Each block is read in anti-clockwise order and value -1 is replaced with 0. It is generated upper binary codes.

1D MP-: Each block is read in clockwise order with -1 replaced with 0. It also generates upper binary codes.

As the preliminary idea arrives from LBP, we also create features using LBP. Based on the LBP, we obtain two different types methods, i.e., (1D MP-LBP(C) and 1D MP-LBP(A)) after using 1D Multi-point Local Binary Pattern (1D MP-LBP).

To show the effectiveness of our proposed feature extraction technique, we need to validate using standard classifier models. For classification, we split the datasets into for training, for testing and for validation purposes. In order to perform classification, we use Artificial Neural Network (ANN) with Backpropagation as our first classifier. In the ANN, two hidden layers are considered for mapping the input to output. The number of nodes in output layer is the number of class labels. In our case, it is 2, i.e., two class labels (Inattentive and Attentive). The sigmoid activation function was used between input-hidden layers. And a softmax activation function was used between the hidden layer and output layer. The loss function used in the error adjustments is the cross-entropy loss function, also known as log loss function. The network was trained for 1000 epochs with learning rate 0.0001. We also use standard classifiers like Decision Trees (DT), Random Forests (RF), Support Vector Machines (SVM), K-Nearest Neighbors (KNN), and state-of-the-art technique LBP [47], and LTP [23].

In KNN, the value of K is varied from and validation-error for each is recorded. The K value corresponding to the minimum error is considered for the metrics evaluation. The Decision Tree classifier uses Gini index for splitting. The number of trees or estimators in Random Forest classifier is taken to be 100, with Gini index as the split criterion. An RBF-kernel-based Support Vector Machine(SVM) Classifier is also used for comparative analysis.

Experimental Results

Two variants of proposed methods, namely 1D MP- and 1D MP- are applied on the dataset collected at our laboratory (IDA LAB, NIT Rourkela). We use the 1D MP-LBP feature selection methods for performance analysis purpose.

We report the performances of classification using proposed technique 1D MP- in Table 1. The results are also summarized with varying block sizes of 5, 7, and 9 in both clockwise and anti-clockwise arrangements.

Table 1

Comparison of performance metrics of proposed model Versus baseline model for 1D MP-

Technique	Block Size	ϑ	Model	F1	Accuracy
1DMP-LTP_TT(C)	Non-Overlapping 5 Segment	α2	ANN	81.08	68.18
			KNN	80.60	70.45
			DT	59.26	50.00
			RF	76.30	63.63
			SVM	81.08	68.18
	Non-Overlapping 7 Segment	α2	ANN	79.45	65.90
			KNN	81.16	70.45
			DT	65.57	52.27
			RF	71.89	59.09
			SVM	77.77	63.63
	Non-Overlapping 9 Segment	α2	ANN	77.77	63.63
			KNN	78.90	65.90
			DT	52.17	50.00
			RF	66.66	52.27
			SVM	77.77	63.63
1DMP-LTP_TT(A)	Non-Overlapping 5 Segment	α2	ANN	82.66	70.45
			KNN	82.35	72.72
			DT	74.62	61.36
			RF	77.14	63.63
			SVM	82.66	70.45
	Non-Overlapping 7 Segment	α2	ANN	87.18	77.27
			KNN	86.84	77.27
			DT	70.17	61.36
			RF	77.77	63.63
			SVM	87.18	77.27
	Non-Overlapping 9 Segment	α2	ANN	82.66	70.45
			KNN	86.84	77.27
			DT	66.83	56.81
			RF	75.52	61.36
			SVM	85.71	75.00

Bold value represents the best result using proposed feature extraction technique with standard classification model

For the 1D MP- technique with a non-overlapping segment of size 5, in terms of F1 and accuracy, DT reports a value of 59.26% and 50.00%, respectively. The random forest outperforms DT. The ANN and SVM report F1 of 81.08% and accuracy of 68.18% each, respectively. However, KNN is better than ANN and SVM in terms of accuracy (70.45%). For block size 7, KNN records the highest F1 (81.16%) and the highest accuracy (70.45%), and DT records the least F1 (65.57%) and least accuracy (52.27%). The KNN outperforms ANN and SVM. Similar observation is seen for block size of 9, KNN is better than ANN, SVM, DT, and RF. The KNN achieves the value of F1 and accuracy 78.90% and 65.90%, respectively. The block length 7 (seven) is found to be most suitable for this dataset. Performance of individual classifier is reported to be best with block length 7.

For the 1D MP- technique with a non-overlapping segment of size 5, the DT is the worst performing classifier (F1 = 74.62%, and accuracy = 61.36%). The RF performs better than DT. The ANN and SVM achieve the highest value of F1 (82.66%). However, KNN outperforms SVM as well as ANN in terms of accuracy (72.72). Similar observation is seen for block size 7, where KNN, ANN and SVM achieve similar results for accuracy. However, ANN and SVM are better than KNN in term F1 (87.18). Similarly, for block size of 9, KNN records the highest F1 (86.84%) and the highest accuracy (77.27%). So, we conclude that 1D MP- technique performs better for SVM and ANN classifier with block size 7.

The performances of classifiers using 1D MP- are displayed in Table 2. We compared our results with the 1D-TP method [23]. Performance of ANN, RF and SVM using 1D MP- as feature extraction technique is reported to be the same. The KNN is found to be the best with F1 (80.55%) and accuracy (68.18%). Performance of classifiers using 1D-TP(A) is found to be better than using 1D-TP(C). Results obtained from four variants of proposed 1D MP- are also displayed. The proposed D MP- is found to be outperforming all other variants as well as 1D MP-.

Table 2

Comparison of performance metrics of proposed model Versus baseline models for LTP

Technique	Model	Block Size	ϑ	F1	Accuracy
1D TP(C) [23]	ANN	Overlapping 9 segment	0.5	79.45	65.90
	KNN			80.55	68.18
	DT			61.53	54.54
	RF			79.45	65.90
	SVM			79.45	65.90
1D TP(A) [23]	ANN	Overlapping 9 segment	0.5	81.08	68.18
	KNN			82.19	70.45
	DT			70.97	59.09
	RF			81.08	68.18
	SVM			81.08	68.18
1D MP-LTPTB-1,0(C)	ANN	Non-Overlapping 9 segment	α2	81.08	68.18
	KNN			84.51	75.00
	DT			67.85	59.09
	RF			81.08	68.18
	SVM			81.08	68.18
1D MP-LTPTB-1,1(C)	ANN	Non-Overlapping 9 segment	α2	85.71	75.00
	KNN			86.10	77.27
	DT			69.09	61.36
	RF			82.19	70.45
	SVM			84.20	72.72
1D MP-LTPTB-1,0(A)	ANN	Non-Overlapping 9 segment	α2	82.66	70.45
	KNN			85.33	75.00
	DT			64.29	54.54
	RF			83.66	72.72
	SVM			84.21	72.72
1D MP-LTPTB-1,1(A)	ANN	Non-Overlapping 9 segment	α2	85.71	75.00
	KNN			86.84	77.27
	DT			80.59	70.45
	RF			83.77	72.72
	SVM			85.71	75.00

Bold value represents the best result using proposed feature extraction technique with standard classification model

The performances of classifier using 1D MP-LBP [47] are shown in Table 3. The ANN classifier using 1D LBP outperforms other classifiers using 1D LBP. The ANN approach outperforms all the models with F1 (78.86%) and accuracy (65.90%). For the 1D MP-LBP(C) method, DT is the worst performing classifier (F1 = 67.74% and accuracy = 54.54%). The ANN and SVM perform better than RF with F1 (81.08%) and accuracy (68.18%) for both. The KNN approach outperforms all other models with F1 (82.19%) and accuracy (70.45%). Likewise, for 1D MP-LBP(A) method, KNN classifier has the highest accuracy (61.36%) and the highest F1 (72.13%) among the models. The proposed 1D MP-LBP(C) is better than 1D LBP.

Table 3

Comparison of performance metrics of proposed model Versus baseline models for LBP

Technique	Model	Block Size	F1	Accuracy
1D-LBP [47]	ANN	Overlapping 7 segment	78.86	65.90
	KNN		68.96	59.09
	DT		63.32	50.00
	RF		70.58	54.54
	SVM		70.58	54.54
1D MP-LBP(C)	ANN	Non-Overlapping 9 segment	81.08	68.18
	KNN		82.19	70.45
	DT		67.74	54.54
	RF		79.45	65.90
	SVM		81.08	68.18
1D MP-LBP(A)	ANN	Non-Overlapping 9 segment	68.65	52.27
	KNN		72.13	61.36
	DT		63.99	59.09
	RF		68.65	52.27
	SVM		68.65	52.27

Bold value represents the best result using proposed feature extraction technique with standard classification model

Average accuracy of popular 1D LBP and our proposed method is displayed in Fig. 6. From this plot, it can be clearly seen that our proposed method is better than 1D LBP.

Fig. 6

Average Accuracy Comparison between 1D MP-LBP and 1D MP-LTP

Discussion

The single-dimensional EEG signals are divided into a number of non-overlapping blocks in the proposed feature extraction technique. Subsequently, each block is converted into ternary (binary) string. Each string is read from clockwise or anti-clockwise. We read the string from both sides to see the effect of position of feature value with in a block. We observed that ternary bits +1 located in lower index contribute very less value if string is read from left while it can contribute significantly if it is read from right. We noticed that the performance of anti-clockwise method is better than clockwise method. It can be noted that the classifiers perform better for 1D MP- while retaining the -1 values in ternary formats.

Accuracy of the classifier using proposed feature extraction technique ranges between 70 and 80%. To further improve the accuracy, one can use other biological signals such as facial expressions, eye gaze, gaze gesture or hand movement which can be collected using web camera or frontal camera. It can be noted that this sensor is not placed on the body part. Placing multiple sensor or multi-channel headset may be a bar for natural attention.

Conclusions and Future Scope

Flipped Learning is found to be an effective learning pedagogy and has been adopted in many higher learning educational institutes across the world. However, major issue with this learning pedagogy is that it cannot identify whether the students pay sufficient attention while watching pre-recorded lecture video.

In this study, we proposed a feature extraction approach to obtain the significant feature for analyzing captured brain signal of students while watching lecture videos. The adapted feature extraction approach is used to classify students involved in a flipped classroom based on the attention level. The results showed that the classifiers combining with proposed 1D MP-LTP method outperform standard classifier as well as LBP-based classifiers. This research can be helpful for instructors to identify students who need special care for improving their learning ability.

This work can be extended in two ways, i) recommend the inattentive lecture video to the students for rewatching, ii) to analyze the multi-source data along with single-channel EEG data in the flipped learning pedagogy.