Support vector machine prediction of obstructive sleep apnea in a large-scale Chinese clinical sample.

STUDY
OBJECTIVES: Polysomnography is the gold standard for diagnosis of obstructive sleep apnea (OSA) but it is costly and access is often limited. The aim of this study is to develop a clinically useful support vector machine (SVM)-based prediction model to identify patients with high probability of OSA for nonsleep specialist physician in clinical practice.
METHODS: The SVM model was developed using the features routinely collected at the clinical evaluation from 6,875 Chinese patients referred to sleep clinics for suspected OSA. Three apnea-hypopnea index (AHI) cutoffs, ≥5/h, ≥15/h, and ≥30/h were used to define the severity of OSA. The continuous and categorized features were selected separately and were further selected through stepwise forward feature selection. The modeling was achieved through fivefold cross-validation. The model discriminative ability was evaluated for the whole data set and four subgroups categorized with gender and age (<65 versus ≥65 years old [y/o]).
RESULTS: Two features were selected to predict AHI cutoff ≥5/h with six features selected for ≥15/h, and six features selected for ≥30/h, respectively, to reach Area under the Receiver Operating Characteristic (AUROC) 0.82, 0.80, and 0.78, respectively. The sensitivity was 74.14%, 75.18%, and 70.26%, while the specificity was 74.71%, 68.73%, and 70.30%, respectively. Compared to logistic regression, Berlin questionnaire, NoSAS Score, and Supersparse Linear Integer Model (SLIM) scoring system, the SVM model performs better with a more balanced sensitivity and specificity. The discriminative ability was best for male <65 y/o and modest for female ≥65 y/o.
CONCLUSION: Our model provides a simple and accurate modality for early identification of patients with OSA and may potentially help prioritize them for sleep study. © Sleep Research Society 2020. Published by Oxford University Press on behalf of the Sleep Research Society.

The prediction models for identifying patients with high probability of obstructive sleep apnea (OSA) reported in previous studies tend to have high sensitivity but low specificity which may lead to overprescription of polysomnography. Moreover, physiological signals or physical examination findings are required in some models which may limit their clinical application. This data mining-driven study proposed a support vector machine (SVM)-based prediction model built with 2, 6, and 6 features commonly collected at clinic visits to identify patients with apnea-hypopnea index (AHI) ≥5/h, ≥15/h, and ≥30/h, respectively. The model was fivefold cross-validated and had a balanced sensitivity (74.14%, 75.18%, and 70.26%) and specificity (74.71%, 68.73%, and 70.30%) which performed better than Berlin questionnaire, NoSAS Score, and Supersparse Linear Integer Model (SLIM) scoring system, particularly for man <65 years old.

Introduction

Obstructive sleep apnea (OSA) is characterized by repeated episodes of upper airway obstruction that results in cessation of airflow during sleep [1]. OSA is a common disease with a prevalence of 9%–38% in the general population [2]. Risk factors for OSA included age, male, obesity, smoking, anomalies of craniofacial features, and menopause in women [3]. Symptoms suggestive of OSA included habitual snore, witnessed apnea, choking or gasping at sleep, frequent awakening, nocturia, unrefreshed sleep, and daytime sleepiness [3]. Early diagnosis of OSA is essential because untreated OSA may increase the probability of developing cardiovascular diseases, metabolic disorders, and neurocognitive dysfunctions [4]. The overnight polysomnography (PSG) is the gold standard for the diagnosis of OSA, and the OSA severity is commonly determined by apnea-hypopnea index (AHI) with cutoff ≥5/h for the presence of OSA, ≥15/h for the presence of moderate–severe OSA, and ≥30/h for the presence of severe OSA [2]. However, PSG is costly and the access is often limited. As a result, prioritizing patients with high risk for moderate–severe OSA for PSG can be crucial for many sleep labs.

A recent meta-analysis showed that care of patients with OSA by nonsleep specialist physician (NSSP) and sleep specialist physician brought similar outcomes in terms of mortality, quality of life, adherence, and symptom score. Since most NSSPs in the included studies had extensive training in sleep medicine, the results may be inferior in the NSSP who were less seasoned or inadequately trained [5]. Hence, the development of a screening model based on clinical features commonly collected at clinic visits to predict the likelihood of OSA would be extremely practical for NSSP. Such a model can also help NSSP to prioritize patients with high pretest probability of OSA for PSG [6].

Prediction models reported in the literature were mostly built using clinical features including demographics (age, gender, smoking, alcohol consumption), comorbidities, anthropometrics, OSA symptoms, physical findings [7], and physiologic measurements (e.g. blood pressure, overnight pulse oximetry, and pulmonary function) [8] collected from either sleep lab- or community-based population. Among prediction models proposed so far, the sensitivity to predict AHI ≥5/h ranged from 66% to 100% while specificity ranged from 30.8% to 76.2%, and the sensitivity to predict AHI ≥15/h ranged from 60.3% to 92.7% while specificity ranged from 33.3% to 90.7% (Supplementary Table S1). The wide-range discriminative ability of models could be attributed to the model complexity, number of participants, prevalence of OSA, and imbalance between different OSA severity proportion. Moreover, most prediction models for OSA tend to have a higher sensitivity with a lower specificity to promote early diagnosis (Supplementary Table S1). These models can potentially cause a high false-positive rate and lead to overprescription of PSG. Nevertheless, some models were established based on the data of which patients with comorbidities were excluded [9], where the generalizability of clinical implication would be constrained.

It is also crucial to validate the model efficacy in subgroups categorized with different features. For example, male patients often have fat distributed to the upper body and a higher percentage of snoring than female patients [10, 11]. Elder patients with OSA may be less susceptible to adverse effects of OSA like sleepiness, impaired quality of life, and mortality compared to middle-aged patients [12, 13]. It is also known that the Asian patients have higher AHI compared to body mass index (BMI)-matched Caucasians due to narrower craniofacial features [14].

Machine learning has been found to be a potential means in addressing these problems by its massive parallelism, self-organization, adaptive learning capability, and robustness. support vector machine (SVM) has been increasingly applied in medical healthcare during the past few years since it can provide systematized architecture for analyzing and extracting important information from complex data [15]. Hence, SVM-based machine learning model may be promising for the prediction of OSA.

The present study aimed to propose an easy-to-use and accurate model to identify patients with OSA at three AHI cutoffs (≥5/h, ≥15/h, ≥30/h). We developed a data mining-driven SVM prediction model using a large-scale sleep lab database with features routinely collected at clinic visits. The model discriminative ability was also tested in the subgroups categorized with gender (men versus women) and age (<65 versus ≥ 65 years old (y/o)). The model discriminative ability was also compared with that of logistic regression, Berlin questionnaire (BQ), NoSAS Score, and Supersparse Linear Integer Model (SLIM) scoring system.

Methods

Data set

The data set developed from information prospectively collected from 7,830 adult patients who underwent initial overnight PSG for the first time in the Center of Sleep Disorder of National Taiwan University Hospital between January 2009 and December 2016. For data mining, only patients who had any following conditions were excluded: non-Chinese (n = 11), total recording time <240 min (n = 7), and missing data (n = 936). A total of 6,875 patients, with 5,223 men and 1,652 women (5,985 < 65 y/o and 890 ≥ 65 y/o), were included (Table 1; Supplementary Table S2).

Table 1.

Comparison of clinical features between patients with and without OSA at three AHI cutoffs

Feature name	Overall (N = 6,875)	AHI < 5 (N = 1,206)	AHI ≥ 5 (N = 5,669)	AHI < 15 (N = 2,664)	AHI ≥ 15 (N = 4,211)	AHI < 30 (N = 4,084)	AHI ≥ 30 (N = 2,791)
Age (y/o)	47.8 ± 14.5	40.6 ± 15.1	49.4 ± 13.9	44.1 ± 15.0	50.2 ± 13.7	46 ± 14.8	50.47 ± 13.8
Man, n (%)	5,223 (76.0)	660 (54.8)	4,563 (80.5)	1,673 (62.8)	3,550 (84.3)	2,798 (68.5)	2,425 (86.9)
BMI (kg/m2)	27.0 ± 5.0	23.5 ± 3.5	27.7 ± 4.9	24.6 ± 3.9	28.4 ± 5.0	25.3 ± 4	29.3 ± 5.3
Neck circumference (cm)	37.7 ± 4.1	34.6 ± 3.4	38.3 ± 4.0	35.6 ± 3.7	39.0 ± 3.9	36.3 ± 3.8	39.7 ± 3.8
Waist circumference (cm)	91.4 ± 13.0	80.9 ± 10.2	93.6 ± 12.5	84.4 ± 10.9	95.8 ± 12.3	86.8 ± 11.2	98 ± 12.7
Current smoker, n (%)	1,104 (16.1)	135 (11.2)	969 (17.1)	334 (12.5)	770 (18.3)	520 (12.7)	584 (20.9)
Alcohol consumption, n (%)	688 (10.0)	80 (6.6)	608 (10.7)	182 (6.8)	506 (12.0)	328 (8)	360 (12.9)
Hypnotic, n (%)	634 (9.2)	151 (12.5)	483 (8.5)	314 (11.8)	320 (7.6)	435 (10.7)	199 (7.1)
Comorbidity
Hypertension, n (%)	2,021 (29.4)	126 (10.4)	1,895 (33.4)	435 (16.3)	1,586 (37.7)	835 (20.4)	1,186 (42.5)
Diabetes, n (%)	580 (8.4)	31 (2.6)	549 (9.7)	117 (4.4)	463 (11)	231 (5.7)	349 (12.5)
CAD, n (%)	248 (3.6)	16 (1.3)	232 (4.1)	62 (2.3)	186 (4.4)	119 (2.9)	129 (4.6)
CHF, n (%)	101 (1.5)	6 (0.5)	95 (1.7)†	22 (0.8)	79 (1.9)	37 (0.9)	64 (2.3)
CVA, n (%)	124 (1.8)	5 (0.4)	119 (2.1)	31 (1.2)	93 (2.2)†	54 (1.3)	70 (2.5)
CKD, n (%)	62 (0.9)	7 (0.6)	55 (1.0)&	13 (0.5)	49 (1.2)†	18 (0.4)	44 (1.6)
COPD, n (%)	67 (1.0)	14 (1.2)	53 (0.9)&	22 (0.8)	45 (1.1)&	37 (0.9)	30 (1.1)&
Asthma, n (%)	490 (7.1)	105 (8.7)	385 (6.8)†	210 (7.9)	280 (6.6)&	321 (7.9)	169 (6.1)†
Hypothyroidism, n (%)	156 (2.3)	25 (2.1)	131 (2.3)&	63 (2.4)	93 (2.2)&	103 (2.5)	53 (1.9)&
Habitual sleep pattern
Habitual SOL (min)	20.9 ± 22.5	25 ± 29.9	20 ± 20.5	23.7 ± 26.7	19.1 ± 19.3	22.6 ± 24.2	18.4 ± 19.6
Habitual SOL < 30 min, n (%)	4,794 (69.7)	770 (63.8)	4,024 (71.0)	1,744 (65.5)	3,050 (72.4)	2,752 (67.4)	2,042 (73.2)
Habitual sleep duration (h)	6.6 ± 3.3	6.6 ± 2.7	6.5 ± 3.5	6.5 ± 2.6	6.6 ± 3.7†	6.5 ± 2.8	6.6 ± 4&
Unrefreshed sleep, n (%)	3,685 (53.6)	770 (63.8)	2,915 (51.4)	1,613 (60.5)	2,072 (49.2)	2,325 (56.9)	1,360 (48.7)
Frequency of awakening in sleep (time/night)	0.5 ± 1.5	0.5 ± 1.5	0.6 ± 1.5&	0.5 ± 1.5	0.6 ± 1.5&	0.5 ± 1.5	0.6 ± 1.5
Awakening at sleep ≥3 times/ night	1,504 (21.9)	237 (19.7)	1,267 (22.3)†	526 (19.7)	978 (23.2)†	802 (19.6)	702 (25.2)
ESS	10.3 ± 4.9	10.0 ± 4.9	10.4 ± 4.9†	9.8 ± 4.8	10.6 ± 4.9	9.9 ± 4.7	11 ± 5
EDS, n (%)	3,764 (54.7)	628 (52.1)	3,136 (55.3)†	1,355 (50.9)	2,409 (57.2)	2,098 (51.4)	1,666 (59.7)
Symptom suggestive of OSA
Snoring, n (%)	5,480 (79.7)	753 (62.4)	4,727 (83.4)	1,912 (71.8)	3,568 (84.7)	3,099 (75.9)	2,381 (85.3)
Witnessed apnea, n (%)	1,066 (15.5)	79 (6.6)	987 (17.4)	221 (8.3)	845 (20.1)	417 (10.2)	649 (23.3)
Frequency of nocturia (times/ night)	1.1 ± 1.2	0.9 ± 1.1	1.2 ± 1.2	1 ± 1.1	1.3 ± 1.2	1 ± 1.1	1.3 ± 1.3
Nocturia ≥2 times/night, n (%)	2,352 (34.2)	308 (25.5)	2,044 (36.1)	746 (28.0)	1,606 (38.1)	1,203 (29.5)	1,149 (41.2)
Witnessed leg jerks in sleep, n (%)	3,278 (47.7)	603 (50)	2,675 (47.2)&	1,303 (48.9)	1,975 (46.9)&	1,974 (48.3)	1,304 (46.7)&
Morning headache, n (%)	799 (11.6)	192 (15.9)	607 (10.7)	351 (13.1)	448 (10.6)†	513 (12.6)	286 (10.2)†
Dry throat at waking up, n (%)	3,856 (56.1)	577 (47.8)	3,279 (57.8)	1,324 (49.7)	2,532 (60.1)	2,132 (52.2)	1,724 (61.8)
AHI (/h)	29.6 ± 26.0	1.9 ± 1.5	35.5 ± 24.9	6.1 ± 4.5	44.5 ± 22.8	11.6 ± 8.7	56 ± 19.5

The data were presented as mean ± standard deviation or number (percentage). CAD, coronary artery disease; CHF, congestive heart failure; CVA, cerebrovascular accident; COPD, chronic obstructive pulmonary disease; CKD, chronic kidney disease. The comparisons between non-OSA and OSA participants were analyzed with the independent t-test and chi-square test. All p-values were <0.001, except for variable marked with & and †, of which the p-values were >0.05 and <0.05, respectively.

Thirty-two clinical features including demographics, anthropometrics, comorbidities, self-reported habitual sleep patterns, and OSA symptoms were collected through self-administered questionnaires and medical records (Table 2). The demographics included age, gender, smoking, alcohol consumption, and hypnotic use defined as taking hypnotics ≥1 time/week over the past month. Anthropometrics included BMI, neck circumference, and waist circumference. Sleep history and OSA symptoms were collected with a self-administered questionnaire (description and definition listed in Supplementary Table S3). Sleep history included unrefreshed sleep, subjective sleepiness, frequency of awakening, awakening ≥3 times/night during sleep, minutes of sleep onset latency (SOL), and hours of sleep duration over the past month. In addition, the SOL < 30 min and the sleep duration categorized as <6 h/day, 6–8 h/day, and ≥8 h/day were added. Subjective sleepiness was assessed by the Sleepiness Scale (ESS) with excessive daytime sleepiness (EDS) defined as ESS ≥ 10 [16]. The OSA symptoms included snore, witnessed apnea, frequency of nocturia, witnessed leg jerks at sleep, morning headache, nocturia ≥2 times/night, and dry throat at wake up.

Table 2.

Thirty-two features input in both SVM model and logistic regression

Demographics	Anthropometric	Comorbidities	Sleep history	Symptoms suggestive of OSA
Age	BMI	Hypertension	SOL (min)	Snore
Gender	Neck circumference	Diabetes	SOL < 30 min	Witnessed apnea
Alcohol consumption	Waist circumference	CAD	Sleep duration, <6, 6–8, and ≥8 h	Frequency of nocturia (time/night)
Current smoking		CHF	Unrefreshed sleep	Nocturia ≥2 times/night
Hypnotics		CVA	Frequency of awakenings at sleep (time/night)	Witnessed leg jerks in sleep
		CKD	Awakenings in sleep ≥3 times/night	Morning headache
		COPD	ESS	Dry throat at waking up
		Asthma	EDS
		Hypothyroidism

CAD, coronary artery disease; CHF, congestive heart failure; CVA, cerebrovascular accident; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease.

The BQ consists of three categories including snore, fatigue, and hypertension, with each category including 2–5 questions with a total of 11 questions. Presence of positive responses to two or more categories indicates high risk for OSA [17]. The NoSAS Score is a 5-item questionnaire which includes neck circumference, BMI, snore, age, and gender. With a range of 0–17, NoSAS Score 4 for neck circumference ≥ 40 cm, scores 3 for BMI 25–30 kg/m2, scores 5 for BMI ≥ 30 kg/m2, scores 2 for snoring, scores 4 for >55 y/o, and scores 2 for male gender. The total score ≥ 8 is considered as high risk for OSA [18]. The SLIM scoring system (size 10) is a 10-feature machine learning model which include age, BMI, diabetes, hypertension, smoking, and female gender. The total points >29 is considered as high risk for OSA [19].

All methods were carried out in accordance with relevant guidelines and regulations. The study protocol was approved by the Research Ethics Committee of NTUH (protocol number 201603113RIND) and the need for consent from participants was waived by the Research Ethics Committee.

Polysomnography

Overnight PSG (Embla N7000, Medcare Flaga, Reykjavik, Iceland) was performed as previously reported [20]. Sleep stages and respiratory events were scored according to the 2007 AASM scoring rule [21]. Apnea was defined as ≥90% decrease in airflow for ≥10 s while hypopnea was ≥30% decrease in airflow ≥10 s associated with ≥4% reduction in arterial oxygen saturation. The PSG parameters collected included sleep efficiency, percentage of slow-wave sleep (% SWS) and % rapid eye movement, AHI, oxygen desaturation index (ODI), percentage of total sleep time with SpO2 < 90% (%TST-SpO2 < 90%), and arousal index (AI).

Feature selection and prediction model design

Figure 1 illustrates the flowchart of SVM prediction model development (Supplementary Table S4 shows the detail of model development). The training procedure of the proposed prediction model includes data input, data exclusion, feature selection, and OSA classification. The training procedure conducted in this study was based only on the training data set to prevent overfitting (i.e. a model that just repeats the samples that it has just seen would have a perfect score but would fail to predict unseen data). Subsequently, a comprehensive blind validation using the testing data set was conducted during the testing stage. In the proposed method, we applied the cross-validation (CV) approach in this study to test the effectiveness of the selected features and the machine learning model. CV is a resampling procedure used to hold out part of the available data as a testing set for model evaluation when data are limited. To perform CV, we put aside a portion of the data not used in model training for testing/validation (Supplementary Figure S1).

Figure 1.

The flow chart of developing SVM-based prediction model. Non-Chinese (n = 11), total recording time <240 min (n = 7), and any missing data (n = 936) were excluded. In the first feature selection stage, we observed that the top half of AUROC and MCC feature ranks were similar in all three different AHI cutoffs suggesting the robustness of these features. Therefore, only features with AUROC or MCC higher than median remained in the model. During the last feature selection, the fewest features were selected to keep AUROC ≥ 0.80.

To optimize the discriminative ability using the fewest features, continuous and categorical features were selected by single-feature SVM Area under the Receiver Operating Characteristic (AUROC) and Matthews Correlation Coefficient (MCC), respectively, during the feature selection [22]. Due to different mathematical characteristics between continuous and categorical features, this study developed a two-stage feature selection procedure to prevent the selection relying on a single type of feature set in the proposed model. In the first stage, only the continuous and categorical features with the top half of AUROC (Table 3) and MCC (Figure 2), respectively, were reserved to reduce the interference from the redundant features, in which these features may be robustly related to different AHI cutoffs. Subsequently, forward stepwise feature selection (FSFS) was exploited in the second half of feature selection. The feature set selected by FSFS was increased stepwise based on the greedy approach [23]. Specifically, the features with the maximum AUROC or MCC in the first stage were regarded as the feature candidates in the second stage. Afterward, each feature from the feature candidates was randomly integrated then used to train a new SVM for evaluating classification performance. During each iteration in the second stage, the add-on feature with the superior SVM performance was reserved for updating the selected feature. Accordingly, the updated selected set was used to evaluate the next incoming feature candidate (Supplementary Table S5). The whole training procedure of FSFS was iterated until the stopping criteria (AUROC ≥ 0.8). To achieve significant clinical application, the selected feature set based on FSFS was aimed to achieve target criteria in AUROC in three AHI cutoffs during the selection procedure. Eventually, the selected feature set after the two-stage feature selection was used to establish the prediction model for OSA recognition based on SVM [24]. The posterior probability of the SVM was used to determine the class of the incoming datum [25], either OSA or non-OSA. To further optimize the classification result, the Youden’s index was employed to find the optimal threshold of SVM posterior probability to determine categories. In addition, the fivefold CV was randomly repeated five times to verify the model reliability. The average AUROC of the fivefold CV for each of the three AHI cutoffs was calculated. The prediction model was trained by sleep lab-based data set with three AHI cutoff 5/h, 15/h, and 30/h, respectively, which means the models based on three AHI cutoffs were fairly trained and validated. To further evaluate the model robustness, the learning curves of three AHI cutoffs were depicted (Supplementary Figure S2).

Table 3.

AUROC of single continuous feature at SVM model for three AHI cutoffs

Order	AHI ≥ 5/h		AHI ≥ 15/h		AHI ≥ 30/h
	Feature	AUROC	Feature	AUROC	Feature	AUROC
1	Waist	0.794	Waist	0.763	Waist	0.754
2	BMI	0.772	Neck	0.747	Neck	0.742
3	Neck	0.769	BMI	0.739	BMI	0.736
4	Age	0.665	Age	0.561	Age	0.584
Features not selected
5	SOL	0.525	Frequency of nocturia	0.520	Frequency of nocturia	0.510
6	Frequency of nocturia	0.513	ESS	0.516	ESS	0.518
7	Frequency of awakening in sleep	0.513	Frequency of awakening in sleep	0.494	Frequency of awakening in sleep	0.503
8	ESS	0.495	SOL	0.488	SOL	0.493

The features were in order according to the AUROC value. In the first stage of feature selection, features with AUROC higher than median were selected.

Figure 2.

MCC values of categorical features correlated with AHI ≥ 5/h, AHI ≥ 15/h, and AHI ≥ 30/h. The feature order was sorted by absolute MCC. The solid bars indicated features with MCC higher than median while the bars with stripe indicate features with MCC lower than median. CKD, chronic kidney disease; CAD, coronary artery disease; CHF, congestive heart failure; CVA, cerebrovascular accident; COPD, chronic obstructive pulmonary disease.

Data analysis

The discriminative ability of the proposed SVM model was evaluated using average of fivefold CV of AUROC, F1 score, accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and positive likelihood ratio (LR+), and negative likelihood ratio (LR−) for each of the three AHI cutoffs. The performance was expressed as mean (95% confidence interval [CI]). The cutoff of AUROC value was identified using the Youden’s index. The SVM model was tested in subgroups categorized by gender (men versus woman) and age (<65 y/o versus ≥65 y/o) to identify subgroups for which the model worked best. The performance of the SVM model was compared to logistic regression, BQ, NoSAS Score, and SLIM scoring system.

For logistic regression, 67% of participants were randomly selected as training set, while the remaining 33% of the participants were selected as testing set. Logistic models with forward selection were used to identify suitable factors to establish the prediction model for AHI ≥5/h, ≥15/h, and ≥30/h in the training test. Each parameter has to be significant at the 0.0001 level to remain in the model. All the remaining variables were listed with odds ratio (OR) and 95% CI. The predictability of AHI ≥5/h, ≥15/h, and ≥30/h was assessed by the AUROC. In addition, sensitivity, specificity, PPV, and NPV were calculated by using the cutoff of Youden’s index.

The clinical features were compared between patients with and without OSA and among four subgroups at three AHI cutoffs. Continuous variables were expressed as mean ± standard deviation (SD) and categorical variables were expressed as percentage. Independent samples t-test and chi-square test were applied as appropriate in comparison of OSA data sets and non-OSA data sets as well as in the subgroups. A two-tailed p-value <0.05 was considered statistically significant. All statistical analyses were conducted by Python (Python Software Foundation. Python Language Reference, version 3.6.1. Available at http://www.python.org), and SAS Version 9.3 (SAS Institute, Cary, NC).

Results

The clinical features of patients are listed in Table 1 and Supplementary Table S2. The mean age was 47.8 y/o and 76% were men. The mean AHI was 29.6/h with a prevalence of 82.5%, 61.3%, and 40.6% at AHI ≥5/h, ≥15/h, and ≥30/h cutoffs, respectively. Compared to patients without OSA, those with OSA were older, more obese, sleepier, and had higher percentage of men, history of smoking and alcohol consumption, comorbidities, and OSA symptoms as well as shorter SOL. The patients without OSA had longer SOL, higher percentage of witnessed leg jerks in sleep, and morning headache than those with OSA. The habitual SOL is weakly correlated with SOL recorded by PSG (Pearson correlation, γ = 0.202, p < 0.001).

Feature selection

The MCCs of categorical features for different AHI cutoffs are listed in Figure 2. Twelve categorical features with top half of MCC value were selected for each AHI cutoff. The results of AUROC evaluation with SVMs trained by each single continuous feature are listed in Table 3. Four continuous features with top half of AUROC in predicting OSA were waist, neck circumference, BMI, and age. In total, 16 features were selected.

In the final feature selection, 2, 6, and 6 features were selected with FSFS for AHI ≥5/h, ≥15/h, and ≥30/h, respectively (Table 4; Figure 3) where the detailed iterations are listed in Supplementary Table S5. The learning curve showed no evidence of overfitting (Supplementary Figure S2). In addition to waist circumference and age, snoring, neck circumference, witnessed apnea, and SOL < 30 min were selected for AHI ≥ 15/h and AHI ≥ 30/h (Table 4). For logistic regression, 7, 10, and 10 features were selected for AHI ≥5/h, ≥15/h, and ≥30/h, respectively (Table 4; Supplementary Table S6). Five features selected in the SVM model including waist circumference, age, neck circumference, snoring, and witnessed apnea were also selected in the LR. The SOL was selected instead of SOL < 30 min. Additional selected features in LR includes BMI, dry throat, gender, hypnotic, and hypertension (Table 4).

Table 4.

The features selected with forward stepwise feature selection of SVM model and logistic regression for three AHI cutoffs

Method	Order	AHI ≥ 5/h	AHI ≥ 15/h	AHI ≥ 30/h
SVM	1	Waist circumference	Waist circumference	Waist circumference
	2	Age	Age	Witnessed apnea
	3		Neck circumference	Age
	4		Snoring	Neck circumference
	5		Witnessed apnea	Snoring
	6		SOL < 30 min	SOL < 30 min
Logistic regression	N/A	Snoring	Snoring	Witnessed apnea
		Gender	Witnessed apnea	Gender
		Age	Dry throat	Snoring
		Neck circumference	Gender	Hypertension
		SOL	Hypnotic	Dry throat
		BMI	Age	Waist circumference
		Waist circumference	Waist circumference	Age
			Neck circumference	Neck circumference
			SOL	BMI
			BMI	SOL

In SVM model, the minimal feature set was selected to achieve the target criteria in the AUROC. When AHI cutoffs were 5/h and 15/h, the target AUROC was set as 0.8. While AHI cutoff was 30/h, the experiment showed that the maximum AUROC was 0.78, so we selected minimum features to achieve the performance. N/A, not applicable.

Figure 3.

This figure illustrates the relationship between AUROC of prediction model and corresponding numbers of features in the stepwise forward feature selection. The results show that fewest numbers of features to achieve AUROC ≥ 0.80 were 2, 6, and 6 for AHI ≥ 5/h, 15/h, and 30/h, respectively. The solid dot and bar indicated mean and standard deviation, respectively. The green circles indicated selected feature numbers with specific AHI cutoff.

Model discriminative ability

The discriminative ability of the SVM model and logistic regression, BQ, NoSAS Score, and SLIM scoring system for three AHI cutoffs are shown in Table 5. The performance of SVM model remains good consistently across three AHI criteria. The AUROC was 0.82, 0.80, and 0.78 for AHI ≥5/h, ≥15/h, and ≥30/h, respectively, while the accuracy was 74.24%, 72.68%, and 70.28%, respectively. The sensitivity was 74.14%, 75.18%, and 70.26%, for AHI ≥5/h, ≥15/h, and ≥30/h, respectively, while the specificity was 74.71%, 68.73%, and 70.30% respectively. Compared to logistic regression, SVM model had similar AUROC and accuracy across three AHI cutoffs. Moreover, at higher AHI cutoffs, the SVM model upheld good sensitivity and NPV without losing specificity and PPV. Compared to BQ, the SVM model had higher AUROC, accuracy, specificity, PPV, and NPV across three AHI criteria while it had higher AUROC, accuracy, sensitivity, and NPV compared to NoSAS Score. Compared to SLIM scoring system, the SVM model had higher AUROC, accuracy, and sensitivity across three AHI criteria.

Table 5.

The performance of SVM, logistic regression, BQ, NoSAS Score, and SLIM scoring system at three AHI cutoffs

Model	AHI cutoff (/h)	Feature no.	AUROC	F1 factor	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	LR+	LR−
SVM	≥5	2	0.82 (0.79–0.85)	0.83 (0.81–0.85)	74.24 (71.59– 76.89)	74.14 (71.33– 76.95)	74.71 (70.88– 78.54)	93.23 (92.17– 94.29)	38.15 (34.79– 41.52)	2.96 (2.52– 3.41)	0.35 (0.30– 0.40)
	≥15	6	0.80 (0.79–0.81)	0.77 (0.74–0.80)	72.68 (70.52– 74.84)	75.18 (67.61– 82.76)	68.73 (61.72– 75.75)	79.32 (76.84– 81.80)	64.03 (59.75– 68.31)	2.45 (2.04– 2.87)	0.36 (0.29– 0.43)
	≥30	6	0.78 (0.77–0.80)	0.66 (0.61–0.70)	70.28 (68.68– 71.88)	70.26 (60.21– 80.31)	70.30 (64.18– 76.43)	61.93 (59.21– 64.35)	77.86 (73.68– 82.03)	2.39 (2.14– 2.64)	0.42 (0.32– 0.52)
LR	≥5	7	0.84 (0.83–0.86)	—	73.77	94.41	37.87	72.55	79.56	1.52	0.15
	≥15	10	0.81 (0.80–0.82)	—	72.14	79.94	62.69	72.21	72.03	2.14	0.32
	≥30	10	0.79 (0.78–0.81)	—	72.83	65.01	78.77	69.94	74.77	3.06	0.44
BQ	≥5	—	0.54 (0.52–0.56)	—	67.58	74.95	32.91	84.01	21.89	1.11	0.76
	≥15	—	0.53 (0.52–0.55)	—	58.39	76.09	30.41	63.34	44.58	1.09	0.79
	≥30	—	0.53 (0.51–0.54)	—	48.09	76.68	28.55	42.31	64.17	1.07	0.81
NoSAS Score	≥5	4	0.70 (0.68–0.71)	—	57.25	50.62	88.39	95.31	27.58	4.36	0.56
	≥15	4	0.68 (0.67–0.70)	—	66.01	57.99	78.67	81.13	54.23	2.72	0.53
	≥30	4	0.68 (0.67–0.69)	—	68.30	64.88	70.64	60.16	74.64	2.20	0.50
SLIM (10 size)	≥5	10	0.69 (0.67–0.70)	0.63	54.68	47.10	90.30	95.80	26.64	4.86	0.59
	≥15	10	0.68 (0.67–0.69)	0.65	64.77	54.33	81.27	82.10	52.96	2.90	0.56
	≥30	10	0.68 (0.67–0.70)	0.62	69.40	62.24	74.29	62.33	74.22	2.42	0.51

The data were presented as mean (95% confidence interval). LR, logistic regression.

The discriminative ability of SVM model in four subgroups are shown in Table 6. The AUROC and accuracy were similar between male and female while AUROC, accuracy, specificity, PPV, and NPV were higher in <65 y/o than ≥65 y/o subgroup. Moreover, the discriminative ability was best for male <65 y/o and modest for female ≥65 y/o. To make this proposed prediction model available to researchers and clinicians, we have built an easy-to-use website (http://howareyou.csie.ntu.edu.tw), which provides OSA probability prediction based on our machine learning model.

Table 6.

The performance of SVM model in subgroups including men, women, <65 y/o, and ≥65 y/o

	AHI cutoff	No. of ≥AHI cutoff	AUROC	F1 score	Accuracy (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	LR+	LR−
Male	≥5	4,653	0.80 (0.77–0.83)	0.82 (0.80–0.83)	71.82 (69.59–74.04)	71.21 (69.04–73.37)	76.06 (73.04–79.08)	95.35 (94.67–96.04)	27.69 (25.55–29.84)	3.00 (2.59–3.42)	0.38 (0.34–0.42)
	≥15	3,550	0.77 (0.77–0.78)	0.76 (0.70–0.81)	69.86 (65.74–73.98)	69.30 (59.97–78.63)	71.08 (63.99–78.16)	83.70 (81.83–85.58)	52.62 (48.17–57.07)	2.44 (2.07–2.82)	0.43 (0.35–0.51)
	≥30	2,425	0.76 (0.74–0.78)	0.66 (0.59–0.73)	69.65 (67.17–72.13)	64.58 (52.25–76.91)	74.06 (67.47–80.64)	68.50 (66.53–70.48)	71.09 (66.21–75.98)	2.52 (2.29–2.74)	0.47 (0.36–0.59)
Female	≥5	1,106	0.78 (0.73–0.83)	0.79 (0.74–0.84)	72.76 (67.28–78.24)	77.75 (70.55–84.96)	62.64 (56.94–68.35)	80.79 (77.74–83.85)	58.64 (50.32–66.96)	2.11 (1.65–2.58)	0.36 (0.23–0.48)
	≥15	661	0.79 (0.75–0.83)	0.67 (0.65–0.69)	67.92 (63.79–72.06)	81.84 (79.15–84.53)	58.64 (50.16–67.12)	57.13 (52.74–61.53)	82.87 (81.88–83.86)	2.02 (1.65–2.39)	0.31 (0.29–0.33)
	≥30	366	0.79 (0.75–0.82)	0.52 (0.47–0.57)	68.17 (61.43–74.91)	77.04 (74.41–79.67)	65.64 (56.66–74.62)	39.47 (33.20–45.75)	90.91 (89.86–91.96)	2.33 (1.68–2.98)	0.35 (0.31–0.40)
<65 y/o	≥5	4,870	0.82 (0.79–0.85)	0.83 (0.80–0.85)	74.52 (71.62–77.41)	74.68 (71.42–77.93)	73.81 (69.80–77.82)	92.56 (91.41–93.71)	40.14 (36.42–43.86)	2.88 (2.46–3.31)	0.34 (0.29–0.40)
	≥15	3,577	0.81 (0.80–0.82)	0.77 (0.74–0.81)	73.43 (71.47–75.39)	76.46 (68.81–84.10)	68.94 (62.42–75.46)	78.66 (76.51–80.82)	66.74 (62.27–71.22)	2.50 (2.15–2.85)	0.34 (0.27–0.41)
	≥30	2,347	0.79 (0.78–0.81)	0.66 (0.62–0.70)	70.96 (69.22–72.70)	71.83 (62.05–81.61)	70.40 (63.93–76.87)	61.22 (58.19–64.25)	79.75 (75.68–83.82)	2.46 (2.13–2.79)	0.40 (0.29–0.50)
≥65 y/o	≥5	799	0.70 (0.64–0.76)	0.78 (0.76–0.79)	65.84 (63.68–68.00)	65.83 (63.37–68.30)	65.73 (52.01–79.45)	94.47 (92.42–96.52)	17.94 (14.85–21.04)	2.19 (0.88–3.50)	0.53 (0.43–0.63)
	≥15	634	0.69 (0.67–0.72)	0.72 (0.66–0.79)	64.94 (59.99–69.88)	65.75 (54.68–76.82)	62.92 (50.00–75.84)	81.77 (77.98–85.57)	42.92 (39.06–46.79)	1.87 (1.33–2.40)	0.54 (0.46–0.63)
	≥30	444	0.70 (0.68–0.72)	0.62 (0.54–0.69)	64.16 (61.20–67.12)	58.52 (45.19–71.85)	69.74 (60.90–78.58)	66.07 (63.15–68.99)	63.23 (58.75–67.71)	1.97 (1.71–2.23)	0.59 (0.47–0.71)
Male < 65 y/o	≥5	3,972	0.81 (0.78–0.84)	0.82 (0.80–0.84)	72.47 (70.56–74.37)	71.80 (69.64–73.97)	76.81 (72.96–80.67)	95.29 (94.52–96.06)	29.46 (27.70–31.22)	3.14 (2.63–3.65)	0.37 (0.33–0.40)
	≥15	3,075	0.78 (0.77–0.79)	0.76 (0.72–0.81)	70.74 (67.41–74.07)	70.18 (61.74–78.62)	71.89 (64.65–79.14)	83.77 (81.54–86.01)	54.50 (50.56–58.44)	2.56 (2.08–3.05)	0.41 (0.34–0.48)
	≥30	2,077	0.77 (0.75–0.79)	0.66 (0.61–0.72)	70.15 (68.40–71.91)	65.96 (54.65–77.26)	73.64 (67.30–79.97)	67.68 (65.67–69.69)	72.62 (68.15–77.10)	2.53 (2.29–2.78)	0.46 (0.35–0.56)
Male ≥ 65 y/o	≥5	591	0.66 (0.50–0.82)	0.77 (0.71–0.84)	64.99 (56.26–73.71)	65.65 (58.02–73.28)	57.09 (30.16–84.02)	94.53 (90.98–98.08)	13.31 (5.67–20.95)	1.82 (0.85–2.79)	0.81 (-0.07–1.70)
	≥15	475	0.69 (0.64–0.75)	0.72 (0.59–0.85)	64.67 (52.79–76.55)	64.21 (46.97–81.45)	66.08 (57.94–74.23)	84.07 (81.07–87.07)	40.99 (30.06–51.92)	1.91 (1.45–2.37)	0.54 (0.31–0.76)
	≥30	348	0.68 (0.65–0.72)	0.60 (0.51–0.70)	61.58 (57.45–65.71)	54.82 (40.88–68.77)	69.49 (61.48–77.50)	67.95 (66.43–69.47)	57.03 (52.80–61.27)	1.80 (1.68–1.92)	0.64 (0.52–0.77)
Female < 65 y/o	≥5	898	0.78 (0.72–0.84)	0.78 (0.74–0.83)	72.32 (67.63–77.01)	78.52 (72.21–84.82)	61.35 (55.96–66.73)	78.23 (75.33–81.13)	62.05 (54.74–69.35)	2.06 (1.67–2.45)	0.35 (0.24–0.46)
	≥15	502	0.79 (0.76–0.83)	0.66 (0.62–0.70)	67.70 (62.00–73.39)	85.67 (81.39–89.94)	57.70 (48.59–66.82)	53.22 (48.26–58.19)	87.83 (84.22–91.43)	2.07 (1.68–2.45)	0.25 (0.17–0.33)
	≥30	270	0.79 (0.76–0.83)	0.49 (0.43–0.55)	66.83 (59.86–73.80)	81.11 (76.69–85.54)	63.44 (55.04–71.83)	35.01 (28.68–41.34)	93.33 (91.53–95.14)	2.30 (1.61–2.99)	0.30 (0.21–0.39)
Female ≥ 65 y/o	≥5	208	0.71 (0.57–0.85)	0.78 (0.72–0.85)	68.88 (62.49–75.26)	68.31 (56.76–79.85)	71.07 (47.64–94.50)	93.19 (89.36–97.02)	29.79 (28.03–31.56)	3.14 (1.14–5.13)	0.44 (0.39–0.49)
	≥15	159	0.68 (0.61–0.74)	0.66 (0.59–0.73)	60.30 (54.38–66.22)	60.32 (50.11–70.54)	60.39 (50.49–70.30)	73.41 (68.65–78.17)	46.04 (39.42–52.66)	1.56 (1.25–1.86)	0.66 (0.49–0.84)
	≥30	96	0.70 (0.65–0.76)	0.59 (0.49–0.69)	66.88 (56.89–76.87)	60.47 (47.36–73.59)	70.95 (55.26–86.63)	58.64 (42.90–74.38)	73.92 (65.99–81.84)	2.66 (0.42–4.91)	0.57 (0.34–0.79)

The data were presented as mean (95% confidence interval).

Discussion

In this study, we proposed a data mining-driven SVM model using a large-scale sleep lab-based data set to predict OSA with three different AHI cutoffs. The features selected in the model were as few as 2, 6, and 6 for AHI ≥5/h, ≥15/h, and ≥30/h, respectively, and all were collected at the clinics. Compared to logistic regression, the SVM model had noninferior discriminative ability, balanced sensitivity and specificity, and with fewer features. The discriminative ability of SVM model was better than BQ, NoSAS Score, and SLIM scoring system. The SVM model worked best for male <65 y/o.

A major strength of our model is that our SVM prediction is built using a large-scale data set from sleep clinics with very few exclusions which enhance the representativeness of the data set and minimize the selection bias of small samples (Supplementary Table S1) [9, 18, 19, 26–36]. Moreover, all 32 features are information routinely collected at the clinic visits and are not physiological parameters derived from overnight pulse oximetry or pulmonary function test. Unlike certain model [7] that includes physical findings of oral cavity which may be difficult to measure precisely [27], we did not include such features for the model development. Similarly, we were concerned that single office blood pressure may not be representative so we did not include it as an input feature.

Another strength of this study is that we designed a two-stage feature selection process with three AHI cutoffs. First-stage feature selection allowed categorical features and continuous features to be selected separately and thus the inclusion of categorical features like witnessed apnea and SOL < 30 min. Furthermore, the FSFS allowed priorities of features to change with different AHI cutoff. Using three AHI cutoffs for OSA would be more stringent than single cutoff. Models that predict AHI ≥ 5/h tend to have high sensitivity but low specificity [27] and could increase the risk of over prescriptions of PSG for those without OSA. Our model achieved a balance in sensitivity and specificity compared to the discriminative ability of the previous studies. When compared to the Duarte’s model [36], which has a high sensitivity and low specificity with two features (Supplementary Table S1), our model has a higher AUROC, sensitivity, and specificity across three AHI cutoffs. The balanced discriminative ability of our model could allow early identification of the patients with high probability of OSA without overprescribing sleep study.

The six features, with the exception of SOL < 30 min, were often selected in the prediction models reported in the literature while SOL < 30 min has never been described to predict AHI ≥ 15/h or AHI ≥ 30/h (Supplementary Table S1). The selection of SOL < 30 min as a feature in the SVM model was a surprise but was echoed by the selection of SOL in logistic regression. Despite that SOL was an input feature in the Boosting model proposed by Caffo et al., it was not selected for their modeling [30]. The difference may be due to the fact that the participants in the Caffo study were from a community population and were older with more females.

Similar to the findings of other studies, neither ESS nor EDS was selected as a model feature, which reflects that there is a high prevalence of EDS in OSA patients regardless of AHI. It may be related to the clinical practice that patients with EDS are more prone to be referred for sleep study and sleepiness is not necessarily concordant with OSA severity [27].

Our results are comparable to another model validated in the Chinese population. Liu et al. [33] developed an SVM-based model to predict OSA with three anthropometric features, i.e. waist size, neck size, and BMI. That study included age, BQ, and anthropometrics as features. The predictability for AHI ≥ 15/h and AHI ≥ 30/h in Liu’s model is highest in females ≤50 y/o. Compared to the discriminative ability for AHI ≥ 15/h in the model reported by Liu et al., our model has higher AUROC and accuracy in elderly males while lower in elderly females. The difference in performance is likely due to the inclusion of OSA symptoms such as snoring and witnessed apnea. Moreover, the age cutoff in our study for subgroup analysis is 65 y/o as opposed to 50 y/o in the study by Liu. We arbitrarily chose 65 y/o as cutoff for subgroup analysis. Elderly OSA patients often have poorer association between AHI and body habitus variables (neck size, BMI, and waist-to-hip ratio), a lower percentage of habitual snoring, and a longer SOL compared to younger patients [5], which may contribute to the poorer performance of our model in the elderly as our model was built with anthropometrics and OSA symptoms.

There are a few limitations in the present study. First, several features highly associated with OSA such as hyperlipidemia and atrial fibrillation were not included in the database and thus not employed for model development. Second, the data set was built form the information collected from the patients referred to our sleep lab for sleep study in whom the prevalence of OSA is high. The result may not be applicable to the general population where the prevalence of OSA is much lower. Third, the participants are all Chinese and the accuracy of this model in other ethnic groups remains unclear. The validity of this model needs to be confirmed in community populations of multiple ethnicity to address the significance and implication. Fourth, using AHI cutoff as the sole target of prediction is one of the limitations of our study. AHI has been known for its loose association with OSA-related outcomes [37] while factors like EDS may have a better prediction of cardiovascular outcome than AHI [38]. In the future, other parameters such as morbidity should be considered as targets of prediction. Fifth, we did not compare our model with STOP-Bang [39] as STOP-Bang was not included as part of our sleep lab routine questionnaires until January 2017. Further study comparing our SVM model with STOP-Bang may be warranted.

In conclusion, our SVM model provides a simple and accurate modality for early identification of patients with OSA. Future studies and machine learning model algorithm development should focus on validation in the sleep lab- and community-based populations with multiethnicity for greater clinical application.

Click here for additional data file.

Click here for additional data file.