Literature DB >> 36097520

Rapid, label-free and low-cost diagnostic kit for COVID-19 based on liquid crystals and machine learning.

Mahboube Esmailpour1, Mohammad Mohammadimasoudi1, Mohammadreza G Shemirani1, Ali Goudarzi1, Mohammad-Hossein Heidari Beni2, Hosein Shahsavarani3,4, Hamid Aghajan2, Parvaneh Mehrbod5, Mostafa Salehi-Vaziri6, Fatemeh Fotouhi5.   

Abstract

We report a label-free method for detection of the SARS-CoV-2 virus in nasopharyngeal swab samples without purification steps and multiplication of the target which simplifies and expedites the analysis process. The kit consists of a textile grid on which liquid crystals (LC) are deposited and the grid is placed in a crossed polarized microscopy. The swab samples are subsequently placed on the LCs. In the presence of a particular biomolecule, the direction of LCs changes locally based on the properties of the biomolecule and forms a particular pattern. As the swab samples are not perfectly purified, image processing and machine learning techniques are employed to detect the presence of specific molecules or quantify their concentrations in the medium. The method can differentiate negative and positive COVID-19 samples with an accuracy of 96% and also differentiate COVID-19 from influenza types A and B with an accuracy of 93%. The kit is portable, simple to manufacture, convenient to operate, cost effective, rapid and sensitive. The simplicity of the specimen processing, the speed of image acquisition, and fast diagnostic operations enable the deployment of the proposed technique for performing extensive on-spot screening of COVID-19 in public places.
© 2022 The Authors.

Entities:  

Keywords:  COVID-19; Diagnostic; Influenza types A and B; Liquid crystals; Machine learning

Year:  2022        PMID: 36097520      PMCID: PMC9452410          DOI: 10.1016/j.biosx.2022.100233

Source DB:  PubMed          Journal:  Biosens Bioelectron X        ISSN: 2590-1370


Introduction

Influenza and coronavirus disease-2019 (COVID-19) are respiratory viral diseases caused by the influenza virus and the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), respectively. They spread easily among people by respiratory droplets/aerosols (The Lancet Respiratory 2020). Both diseases produce similar symptoms, making their differentiation at the symptomatic diagnosis level a challenge. Most people affected by COVID-19 demonstrate mild to moderate respiratory symptoms and some may have no symptoms (asymptomatic infection). Others develop severe symptoms such as difficulty in breathing, shortness of breath and chest pain, and may require specialized treatment and intensive care (Ahmad et al., 2020). Another acute respiratory illness caused by influenza viruses is seasonal flu (Adeola et al., 2015). The two types of flu viruses, namely influenza viruses type A and type B, are biologically similar to each other and are undetectable in size and shape (Song et al., 2016). The influenza virus type B mainly infects only humans (Bouvier and Palese 2008) whereas the type A virus infects birds, pigs, dogs, and other animals as well as human beings (Song et al., 2016). Laboratory diagnosis of influenza and COVID-19 is routinely conducted by targeting genetic materials of the virus in respiratory specimens (Zheng et al., 2020). The most common assays for the detection of viral genomes are nucleic acid amplification tests (NAATs) (Shen et al., 2020) such as real-time reverse-transcription polymerase chain reaction (rRT-PCR) (Pang et al., 2020; Torabi et al., 2020; Kapoor et al., 2021). Despite the fact that rRT-PCR has been rolled out as the standard diagnosis method (Balboni et al., 2012) for COVID-19 (Pang et al., 2020; Yang et al., 2020) and routinely administered nowadays, its cost and the required time for the PCR process render it less accessible for managing the unprecedented increase in the number of needed daily tests for large-scale epidemiological surveillance, infection control and treatment demands of a pandemic. A biosensor consists of a set of instruments with specific biochemical reactions which may interact electrically, optically, thermally, or chemically with the target material or the environment to detect isolated enzymes, tissues, cells, or a chemical element. When an array of liquid crystals (LC) is placed between two crossed polarizers, the presence of an external material in their vicinity can make the LC molecules reorient according to their proximity to the different parts of the material. As LCs are anisotropic in their natural state (Li and Alivisatos 2003; Woltman et al., 2007; Hähsler et al., 2020; Prakash et al., 2020), their reorientation due to the external material leads to the appearance of specific patterns on the surface of one polarizer when light is emitted to the sample from the side of the other polarizer. The pattern can be recorded and analyzed in order to infer the composition of the external material. Based on this characteristic, a variety of sensors based on LCs have been introduced (Su et al., 2019; Hong and Jang 2020). Common methods for LC-aqueous interfaces use glass substrates (Khan and Park 2014; Munir and Park 2018). The glass substrates are usually coated with N,N-dimethyl-N-octadecyl-3aminopropyltrimethoxysilyl chloride (DMOAP) (Kahn 1973; Chen and Yang 2013; Chen et al., 2019) or octadecyltrichlorosilane (OTS) (Brake et al., 2003a, Brake et al., 2003b) to align liquid crystals homotropically. Mesh grids made of gold or copper are placed on the coated substrates (Li and Alivisatos 2003; Hartono et al., 2008; Wang et al., 2015; Wang et al., 2020). Due to the presence of substrates and the required alignment between them and the polarizers, implementing this method requires precision tools which would make its fabrication expensive and not suitable for large-scale roll-out. In this work, we designed a substrate-free technique developed for the detection of the SARS-CoV-2 virus in nasopharyngeal swab samples. We used textile grids instead of copper grids (Green et al., 2019) without using any glass or polymer material as substrate for anchoring the LCs. This also means that the presence of air both above and below the LCs causes their homotropic alignment. The detection of target biomolecules is achieved without the use of any agents. An important feature of our LC-based detection approach is that the virus could be killed by UV light right after specimen is obtained in order to prevent any contamination of operators in the lab. In the developed method, the specimen does not need to undergo purification steps and no multiplication of the target biomolecule is necessary. In order to capture variations within each set of positive and negative samples caused by the presence of various materials in the specimen, the developed technique employs machine learning modules which are trained using a set of positive and negative samples labelled by expert clinicians. Images of the patterns created by the deposition of the specimen on the detection kit are processed by machine learning functions that are trained to extract particular features from the collection of the two sample types which best distinguish them. Once the detection system is trained, performing diagnosis on a new sample can be instantaneous using an image acquired from the specimen prepared in a similar fashion. The simplicity of the specimen processing and the speed of image acquisition and diagnosis operations enable the deployment of the proposed technique for performing extensive on-spot screening of COVID-19 in places such as border entries and airports. In addition to the main model trained for labelling the incoming samples as COVID-19 positive or negative, a separate model has been trained in the same framework to differentiate COVID-19 samples from influenza types A and B samples. The kit has advantages over other biosensor-based techniques as it supports a label-free method (requiring no chemical tagging of the target biomolecules), and offers a portable, highly sensitive, easy to use and cost-effective solution to large-scale screening demands of a pandemic.

Experimental setup

Sensor preparation

An overview of the sensing and processing flow is shown in Fig. 1 . The kit was constructed using an autoclaved textile agitated with 0.02 μl of E7 as LC molecules (Vallamkondu et al., 2018). The LCs in the grid holes of the textile are aligned homotropically due to the presence of air above and below them (Hartono et al., 2008; Popov et al., 2017). Due to the homotropical orientation of the LCs, the incident light experiences ordinary refractive indices and does not get retarded. The beam of light entering from one polarizer side cannot pass through the second crossed polarizer due to the presence of the LCs at random angles and thus the pattern appearing on the other surface will be dark. In the presence of a particular biomolecule, the direction of LCs changes locally based on the morphological properties of the biomolecule and a pattern is formed on the surface of the output polarizer as the incident light experiences various effective refractive indices at neighboring locations due to the local reorientation of the LCs. The orientation response depends on the properties of the deposited molecules such as polarity and morphology, and the resulting patterns can be different for different molecules. The orientation of the LCs near the surface is known to be dependent on the nanoscopic structure and chemical properties of the surface. Anchoring of LCs on the surface after presenting the specimen is influenced by the external structure of the biomolecules present in the specimen and thus the resulting pattern will depend on the structure of these biomolecules (Jang et al., 2006). For example, several structural differences have been reported between the influenza A and B viruses and the SARS-CoV-2 virus (Terrier et al., 2021), a list of which is included in Table 1 .
Fig. 1

An overview of the method to diagnose COVID-19, influenza types A and B with the proposed technique based on the reorientation of LCs caused by the addition of the specimen.

Table 1

Structural and compositional characteristics of the viruses tested in this study.

Influenza AInfluenza BSARS-CoV-2
ClassificationOrthomyxoviridaeOrthomyxoviridaeCoronaviridae
Size and morphology80–120 nmSpherical or filamentous80–120 nmSpherical or filamentous50–140 nmSpherical
GenomeSingle-stranded, negative-sense RNA-segmented (13.5 kb)Single-stranded, negative-sense RNA-segmented (13.5 kb)Single-stranded, positive-sense RNA-linear (29.9 kb)
Integral proteinHA, NA, M2HA, NA, NB, and BM2S, E, M
Isoelectric range (pH)6.5–76.5–75.2–6.2
NP protein length498 amino acids560 amino acids419 amino acids
An overview of the method to diagnose COVID-19, influenza types A and B with the proposed technique based on the reorientation of LCs caused by the addition of the specimen. Structural and compositional characteristics of the viruses tested in this study.

Biological experiments

To conduct the tests, upper respiratory specimens were collected using swabs and diluted in a viral transport medium (VTM). Nowadays, collecting upper respiratory samples is performed routinely and such samples are used in the procedure of PCR-based COVID-19 and influenza diagnosis. Therefore, the proposed method does not introduce any additional burden for biological sample acquisition.

Influenza samples

Real-time PCR reactions were performed for the influenza samples using Rotor-Gene Q (QIAGEN). The reaction mixture consisted of Master Mix, SuperScriptIII RT/Platinum Taq Mix, specific primers and fluorescent probes. The thermal cycling program was performed based on the WHO guideline (Mehrbod et al., 2017). Briefly, reverse transcription was carried out at 50 °C for 30 min, followed by 95 °C for 2 min, and amplification at 95 °C for 15 s followed by 55 °C for 30 s.

COVID-19 samples

Respiratory specimens (Oropharyngeal/Nasopharyngeal swabs) from COVID-19 suspected cases who were referred to the COVID-19 National Reference Laboratory, Pasteur Institute of Iran, were included in the study. SARS-CoV-2 infection was tested by Real-Time PCR as described previously (Salehi-Vaziri et al., 2021). Briefly, RNA extraction was carried out by a QIAcube HT system with a QIAamp 96 Virus QIAcube HT Kit according to the manufacturer's instructions. Real-Time PCR assay was done using 2019-nCoV Nucleic Acid Diagnostic kit (Sansure Biotech, Changsha, China) according to the manufacturer's protocol.

Preprocessing of acquired images

The images acquired from the detection kit after the deposition of the specimen were of slightly different sizes due to the slight variations of imaging systems used in separate laboratory setups to prevent cross contamination of samples. For employing machine learning techniques, all images used as input should be of the same size. Therefore, the acquired images were first resized to pixels after an anti-aliasing filter and a bilinear interpolation were applied to them. To remove corner effects caused by imaging and also reduce the dimensionality of the input space in order to decrease the risk of overfitting, all images were cropped from the center so the final images were of the size pixels. Finally, the resulting images were visually compared with the original images and no significance differences were observed in their quality. Examples of the application of preprocessing steps on a few input images can be found in the supplementary information Fig. A2.

Model training

A common issue in the classification of medical images is the lack of adequate training samples due to the cost of collecting medical data. This issue is more severe in the training of deep learning models as a large number of samples are often needed in training such networks. In these scenarios, a known strategy that alleviates the need for extensive data collection is Transfer Learning (TL), in which a network is pre-trained on a large pre-existing dataset and used as a feature extractor module for the intended application. Then, a classifier module is designed and trained on the target data in combination with the pre-trained feature extractor. There are two options for training such combined models: Only the parameters of the classifier module are learnable. The parameters of both the classifier and feature extractor modules are learnable. The first option was used in this study as it supports a lower computational cost and yielded better results on our datasets. For the feature extractor module, the Alexnet neural network (Krizhevsky et al., 2012) with fixed parameters, which are available as open-access on the internet, was used (see Fig. 2 a). For the classifier module, four fully connected layers were used. As shown in Fig. 2, the output of the feature extractor module is a Tensor of the size (6, 6, 256). Therefore, a flattened layer should be used to contain the extracted features, the output of which would be of the size (9216, 1). The feature vector is passed onto the fully connected layers of the classifier module (see Fig. 2b), which produce an output vector of size 2. The first and second components of the output are, respectively, the membership degrees of the input image to the first and second classes. To train our model, a softmax layer was applied to the classifier output vector in order to have a probability vector of size 2 defined in equation (1):where is the component of the vector , the output of the classifier. The following loss function, expression (2), was used for training the network:where is the label of the input image, is the predefined cost of misclassification of the input image in class , is the output probability vector of the input image, is the regularization coefficient, is the set of classifier layers and is the matrix parameter of the layer in the classifier model with the dimension , and and are respectively the dimensions of the input and output of the classifier layer. is the Frobenius norm of matrix . The regularization term reduces the complexity of the model by preventing the model parameters from freely changing and causing the model to overfit on the training data.
Fig. 2

Deep learning neural network. a) Feature extractor module. b) Classifier module.

Deep learning neural network. a) Feature extractor module. b) Classifier module. The method of assigning a weight to each loss term related to the misclassification of the input image in each class is a commonly used practice when the training dataset is unbalanced. In fact, in the case of misclassification of an image with, let's say, the true label of zero, the first component of the probability vector, , obtained from the output network (see equation (1)) is lower than its second component. Therefore, the term can be used as a measure for the cost associated with misclassifying the input image with a true label of zero. On the other hand, if the number of images with true labels of zero is low, setting to a higher value than not only associates more cost to the model for misclassifying these images, but it also increases the gradient (rate of change) of the model parameters. As a result, misclassifying images with the true label of zero makes the model to face a higher cost and forces it to change its parameters more rapidly due to higher gradient values. In other words, the higher value for the weight tries to make a fair balance between the changes caused in the model by images with label 1 due to their higher number, and the changes applied to the model by images with label 0 due to the higher value of . To set values for the regularization term and the weight vector , we use Bayesian optimization (see supplementary information). Finally, to use the model for labeling an input image, we use equation (3):

Results

The molecular interaction between surfactants and LCs has been stipulated in earlier studies as the reason for the LC reorientation (Brake et al., 2003a, Brake et al., 2003b; Brake et al., 2003a, Brake et al., 2003b). The theory of biomolecular interactions (Mukherjee 2014) and the elastic effect have also been investigated (Lee et al., 2017). More recently, the bond between proteins and LCs that causes the reorientation of LCs was explained by the Raman phenomenon (Kalita et al., 2022). In our setup, adding the specimen to the detection kit causes a change in the appearance of the pattern due to the new orientation of the LCs. The process is filmed with a fixed camera mounted on a regular lab microscope and the video is used for further processing. Fig. 3 shows the microscopic images of the LCs with air, influenza A virus, influenza B virus, COVID-19 virus, and VTM deposited on the textile's 300 μm 300 μm grid square. The images are captured after 1 min of specimen deposition. The captured pattern is dark in the air/LC/air sample as the LCs are oriented homotropically whereas varying colored patterns are observed for the molecules/LC/air samples. The patterns can be differentiated from each other by naked eyes, but variations also exist within each set of samples. The zoomed versions of Fig. 3a and b schematically indicate the orientation of the LCs within the grid square from the top view.
Fig. 3

Polarized microscopic images of LCs in the textile grid squares in presence of a) Air, b) Influenza A virus, c) Influenza B virus, d) COVID-19 virus, and e) VTM. The schematic zoomed versions in (a) and (b) illustrate the LC orientation from the top view.

Polarized microscopic images of LCs in the textile grid squares in presence of a) Air, b) Influenza A virus, c) Influenza B virus, d) COVID-19 virus, and e) VTM. The schematic zoomed versions in (a) and (b) illustrate the LC orientation from the top view.

Classification results

We used five different image sets each containing samples from a condition as follows, and all acquired by the imaging system described earlier: COVID-19-P (P): Positive COVID-19. COVID-19-N (N): Negative COVID-19 (no further information is provided regarding infection with FLU-A or FLU-B). FLU-A (A): Positive influenza type A. FLU-B (B): Positive influenza type B. (C): Negative FLU-A and FLU-B (no further information is provided regarding infection with COVID-19). The number of image samples in each of these five sets of data is listed in Table 2 . Using these image sets, four separate binary classifiers were trained. It should be mentioned that each classifier tries to distinguish between individuals affected with a particular virus and the ones not affected by it. Therefore, as indicated in Table 3 , four classifiers were trained based on different groupings of the mentioned image sets. The test accuracy results of the classifiers with their standard deviations are also listed in Table 3, and the confusion matrices are shown in Fig. 4 . Expectedly, the performance of the model for the classifier P vs. N is superior to that of the other classifiers as in other cases multiple types of samples and consequently different types of patterns are present in one of the two classes used for training the classifier.
Table 2

The number of image samples in each of five image sets. P, N, A, B and C respectively stand for Positive COVID-19, Negative COVID -19, FLU-A, FLU-B and Negative FLU-A and FLU-B image sets.

Image sets# of images
P1434
N491
A1050
B946
C373
Table 3

The number of image samples and test accuracy of the four binary classifiers with their standard deviations.

Binary classifier# of train samples# of test samplesMean test accuracyBest accuracy
P vs. N106726795.17±0.7996%
P vs. A, B, N232958387.28±2.6893%
A vs. P, B, C205651581.25±5.1690%
B vs. P, A, C304276185.42±1.5289%
Fig. 4

Confusion matrices of different classifiers trained on the image sets (for the best accuracies reported in Table 3). a) Confusion matrix of COVID-19-P (P) vs. COVID-19-N (N), b) Confusion matrix of COVID-19-P (P) vs. FLU-A, FLU-B, COVID-19-N (ABN), c) Confusion matrix of FLU-A (A) vs. COVID-19-P, FLU-B, C (PBC), d) Confusion matrix of FLU-B (B) vs. COVID-19-P, FLU-A, C (PAC).

The number of image samples in each of five image sets. P, N, A, B and C respectively stand for Positive COVID-19, Negative COVID -19, FLU-A, FLU-B and Negative FLU-A and FLU-B image sets. The number of image samples and test accuracy of the four binary classifiers with their standard deviations. Confusion matrices of different classifiers trained on the image sets (for the best accuracies reported in Table 3). a) Confusion matrix of COVID-19-P (P) vs. COVID-19-N (N), b) Confusion matrix of COVID-19-P (P) vs. FLU-A, FLU-B, COVID-19-N (ABN), c) Confusion matrix of FLU-A (A) vs. COVID-19-P, FLU-B, C (PBC), d) Confusion matrix of FLU-B (B) vs. COVID-19-P, FLU-A, C (PAC).

Conclusion

In summary, an inexpensive, safe, simple-to-use, and portable biosensor based on the nomatic liquid crystal, commercially available textile, and the use of machine learning techniques for pattern classification has been developed. The initial orientation of the liquid crystals inside the textile grid pixels is homotropic, and the addition of biological specimen causes the orientation of LCs to change, resulting in a pattern which contains features specific to the structure and morphology of the biomolecules present in the specimen. The developed biosensor was tested to differentiate between influenza types A and B and the COVID-19 viruses. The in-vitro method proposed in the current study provides a low-cost, convenient, and rapid diagnostic platform which can be used in clinical diagnosis of infectious diseases, and in particular fast-screening of persons suspected to carry COVID-19 or Flu viruses. The label-free detection method employed in the proposed technique simplifies the analysis process. We have designed a simple protocol for data acquisition, which can be also deployed using a mobile phone for capturing the specimen images and transferring them to a processing server. The method is quantitative and can detect the presence of specific molecules or quantify their concentration in the medium using machine learning techniques. Since the proposed method does not require multiplication of the virus for its detection function, the specimen can be treated by UV at the time of sample acquisition to kill the viruses and reduce the risk of contamination in further steps of analysis.

Ethics

According to the principles of research ethics relating to conducting research on extra samples, the use of additional and extra samples of participants may not have any issues for ethical evaluation considerations in such cases when it is not possible to recall and obtain informed consent from the participants who had referred to clinics on an outpatient and temporary basis, and when no results would be made available to the sample owner and the research results would not affect the sample owner in any way. No characteristics of the sample owners were available to the researchers involved in this project and all samples were identified only by codes throughout the analysis.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CRediT authorship contribution statement

Mahboube Esmailpour: Investigation, Validation, Writing – original draft, (liquid crystals). Mohammad Mohammadimasoudi: Conceptualization, Supervision, (liquid crystals), Writing – review & editing. Mohammadreza G. Shemirani: Investigation, Methodology, Validation, Writing – original draft, (liquid crystals). Ali Goudarzi: Investigation, Methodology, Validation. Mohammad-Hossein Heidari Beni: Software, Methodology, (machine learning), Data curation, Formal analysis, Writing – original draft, (machine learning). Hosein Shahsavarani: Supervision, (biology), Investigation. Hamid Aghajan: Supervision, (machine learning), Methodology, (machine learning), Writing – review & editing. Parvaneh Mehrbod: Resources, (COVID-19 samples, influenza types A and B samples), Investigation, Writing – original draft, (biology). Mostafa Salehi-Vaziri: Resources, (COVID-19 samples), Writing – review & editing, (biology). Fatemeh Fotouhi: Resources, (COVID-19 samples, influenza types A and B samples), Methodology, Writing – review & editing, (biology).

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
  27 in total

1.  Anchoring of nematic liquid crystals on viruses with different envelope structures.

Authors:  Chang-Hyun Jang; Li-Lin Cheng; Christopher W Olsen; Nicholas L Abbott
Journal:  Nano Lett       Date:  2006-05       Impact factor: 11.189

Review 2.  Liquid-crystal materials find a new order in biomedical applications.

Authors:  Scott J Woltman; Gregory D Jay; Gregory P Crawford
Journal:  Nat Mater       Date:  2007-11-18       Impact factor: 43.841

3.  Label-free liquid crystal immunosensor for detection of HBD-2.

Authors:  Xiuxia Su; Wenjing Huo; Dong Yang; Chonglin Luan; Jia Xu
Journal:  Talanta       Date:  2019-05-18       Impact factor: 6.057

Review 4.  The biology of influenza viruses.

Authors:  Nicole M Bouvier; Peter Palese
Journal:  Vaccine       Date:  2008-09-12       Impact factor: 3.641

5.  Liquid crystal-based proton sensitive glucose biosensor.

Authors:  Mashooq Khan; Soo-Young Park
Journal:  Anal Chem       Date:  2014-01-16       Impact factor: 6.986

6.  Label- and enzyme-free detection of glucose by boronic acid-coupled poly(styrene-b-acrylic acid) at liquid crystal/aqueous interfaces.

Authors:  Sundas Munir; Soo-Young Park
Journal:  Anal Chim Acta       Date:  2018-05-15       Impact factor: 6.558

7.  Association of IFITM3 rs12252 polymorphisms, BMI, diabetes, and hypercholesterolemia with mild flu in an Iranian population.

Authors:  Parvaneh Mehrbod; Sana Eybpoosh; Fatemeh Fotouhi; Hadiseh Shokouhi Targhi; Vahideh Mazaheri; Behrokh Farahmand
Journal:  Virol J       Date:  2017-11-09       Impact factor: 4.099

Review 8.  Road toward rapid-molecular point of care test to detect novel SARS-coronavirus 2019 (COVID-19): Review from updated literature.

Authors:  S Ahmad; N Ali; M Kausar; H Misbah; A Wahid
Journal:  Allergol Immunopathol (Madr)       Date:  2020-07-01       Impact factor: 1.667

9.  Aptamers, the bivalent agents as probes and therapies for coronavirus infections: A systematic review.

Authors:  Raheleh Torabi; Reza Ranjbar; Mehrdad Halaji; Mohammad Heiat
Journal:  Mol Cell Probes       Date:  2020-07-04       Impact factor: 2.365

10.  An Open Receptor-Binding Cavity of Hemagglutinin-Esterase-Fusion Glycoprotein from Newly-Identified Influenza D Virus: Basis for Its Broad Cell Tropism.

Authors:  Hao Song; Jianxun Qi; Zahra Khedri; Sandra Diaz; Hai Yu; Xi Chen; Ajit Varki; Yi Shi; George F Gao
Journal:  PLoS Pathog       Date:  2016-01-27       Impact factor: 6.823

View more

北京卡尤迪生物科技股份有限公司 © 2022-2023.