Case-Ascertainment Models to Identify Adults with Obstructive Sleep Apnea Using Health Administrative Data: Internal and External Validation.

BACKGROUND: There is limited evidence on whether obstructive sleep apnea (OSA) can be accurately identified using health administrative data. STUDY DESIGN AND METHODS: We derived and validated a case-ascertainment model to identify OSA using linked provincial health administrative and clinical data from all consecutive adults who underwent a diagnostic sleep study (index date) at two large academic centers (Ontario, Canada) from 2007 to 2017. The presence of moderate/severe OSA (an apnea-hypopnea index≥15) was defined using clinical data. Of 39 candidate health administrative variables considered, 32 were tested. We used classification and regression tree (CART) methods to identify the most parsimonious models via cost-complexity pruning. Identified variables were also used to create parsimonious logistic regression models. All individuals with an estimated probability of 0.5 or greater using the predictive models were classified as having OSA.
RESULTS: The case-ascertainment models were derived and validated internally through bootstrapping on 5099 individuals from one center (33% moderate/severe OSA) and validated externally on 13,486 adults from the other (45% moderate/severe OSA). On the external cohort, parsimonious models demonstrated c-statistics of 0.75-0.81, sensitivities of 59-60%, specificities of 87-88%, positive predictive values of 79%, negative predictive values of 73%, positive likelihood ratios (+LRs) of 4.5-5.0 and -LRs of 0.5. Logistic models performed better than CART models (mean integrated calibration indices of 0.02-0.03 and 0.06-0.12, respectively). The best model included: sex, age, and hypertension at the index date, as well as an outpatient specialty physician visit for OSA, a repeated sleep study, and a positive airway pressure treatment claim within 1 year since the index date.
INTERPRETATION: Among adults who underwent a sleep study, case-ascertainment models for identifying moderate/severe OSA using health administrative data had relatively low sensitivity but high specificity and good discriminative ability. These findings could help study trends and outcomes of OSA individuals using routinely collected health care data.

Introduction

Obstructive sleep apnea (OSA) is the most common respiratory sleep disorder, afflicting up to one billion people globally.1–3 Untreated OSA is a potentially important modifiable risk factor for adverse acute health outcomes associated with greater healthcare utilization.4–9 While consistent evidence exists on the association between OSA and increased all-cause and cardiovascular mortality, the influence of OSA on other health outcomes has been less clear.10 Therefore, larger real-world generalizable population-based studies that follow participants for longer periods of time are required to investigate outcomes, treatment, epidemiology, and medical care use for individuals with OSA.7

A referral to a sleep physician followed by in-laboratory polysomnography (PSG), an overnight sleep study in a sleep laboratory, is generally considered the reference standard for the diagnosis of OSA ().11 The term PSG implies the recording, analysis, and interpretation of multiple physiologic signals collected simultaneously, including but not limited to an electroencephalogram, electromyogram, electrooculogram, electrocardiogram, and respiratory signals.12 The apnea–hypopnea index (AHI), one of the respiratory signals derived from the PSG, which calculates the number of times an individual partially or completely stops breathing per hour of sleep, is used to diagnose OSA and measure its severity.13 Based on the AHI, OSA may be categorized as mild (AHI of 5–14.9), moderate (AHI of 15–30), or severe (AHI >30) OSA.14 Treatments for OSA include conservative measures (weight reduction and positional therapy), positive airway pressure (PAP) treatment, oral appliance treatment (OA), and surgery.14,15 PAP treatment is the recommended modality for moderate to severe OSA in adults.15 Follow-up sleep studies may be used to titrate PAP or OA to individual needs or to reassess the severity of OSA following surgery, weight reduction, or on positional therapy.

If we can accurately ascertain individuals with OSA from health administrative data, needed epidemiological studies could be conducted. However, PSG results on OSA presence and severity are not typically collected on a population-level, and OSA may not be accurately captured by diagnostic codes,16 making a valid interpretation of the study results problematic. Without valid case ascertainment algorithms, studies that identify OSA in administrative data17–23 using only diagnostic codes are at substantial risk of non-quantifiable misclassification bias.

A few prior studies have examined health administrative codes for identifying OSA in health administrative data.24,25 The best case-ascertainment algorithms derived from those studies were 1) a combination of a billing claim for a PSG and receipt of a PAP device in a 2-year period, which yielded a sensitivity and specificity of 19% and 98%, respectively, to identify individuals with OSA based on diagnosis by a sleep physician or AHI ≥524 and 2) a combination of one hospital discharge code or three outpatient billing claims for OSA in a 2-year period, which yielded a sensitivity and specificity of 12% and 82%, respectively, to identify individuals with OSA (AHI ≥5).25 Additional consideration of hypertension, diabetes, myocardial infarction, or stroke in the case definition improved specificity to 91%, while the sensitivity remained the same.25 Together, these findings suggest that no code or combination of codes achieves an adequate level of accuracy to confidently rule in or rule out the presence of OSA.26 These studies, however, considered only a limited number of variables in tested algorithms, mostly based on diagnoses or procedural codes, and were limited by a lack of internal and external validations. Multivariable modelling techniques may potentially improve the accuracy of case ascertainment algorithms.27,28

This study aimed to address the limitations of previous research by developing multivariable case-ascertainment models based on demographics, comorbidities, physician characteristics, diagnostic tests, and procedure codes from health administrative data to accurately identify OSA among individuals with a high enough pre-test probability of OSA to be referred for a sleep study.

Methods

Study Design

We conducted a retrospective cohort study to derive and validate a case-ascertainment model to identify adults with OSA in health administrative data by linking clinical data from diagnostic sleep studies from two academic sleep centers in Ontario (the most populous province in Canada) and provincial health administrative databases housed at ICES (formerly, the Institute for Clinical and Evaluative Sciences).

This project was approved by the Research Ethics Boards of the Ottawa Health Science Network, the Western University Health Sciences, and the Sunnybrook Health Sciences Centre (Canada). ICES is an independent, non-profit research institute whose legal status under Ontario’s health information privacy law allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement.

Data Sources

Clinical Data

To derive and internally validate predictive models, we utilized The Ottawa Hospital (TOH) Sleep Database (NCT03834792), which contains clinical and PSG data on all consecutive adults who underwent a diagnostic sleep study in the TOH Sleep Center between 2015 and 2017. To validate models externally, we used the London Health Sciences Centre Sleep and Apnea Assessment Unit PSG database, which includes clinical and PSG data on all consecutive adults who underwent a diagnostic sleep study in the London Health Sciences Centre between 2007 and 2015. Details on databases, including scoring criteria used and data collection, are presented in the ).

Provincial Health Administrative Data Housed at ICES

We deterministically linked clinical databases to provincial health administrative databases using encrypted individual health card numbers. Residents of Ontario have universal public health insurance under the Ontario Health Insurance Plan (OHIP), with sleep medicine services being 100% reimbursed. Since 1991, ICES has housed high-quality individual-level administrative datasets on publicly funded services provided.29,30

Databases used in this study included the Registered Persons Database (RPDB) [demographic information]; Discharge Abstract Database (DAD) [information on hospital admission]; Same Day Surgery Database (SDS) [same day surgery information]; the National Ambulatory Care Registry System (NACRS) [information on emergency visits]; the OHIP database [physician billing and technical fees for procedures including PSGs]; the Canadian Census [neighborhood socioeconomic details]; ICES Physician Database (IPDB) [information on physician specialty]. We also utilized ICES-derived disease-specific databases. Finally, for all insured Ontario residents who have been diagnosed with OSA by a sleep physician registered with the Assistive Devices Program (ADP), funding is provided for PAP systems and recorded in the ADP database.31 The databases listed were linked using unique encoded identifiers. A detailed description of the ICES datasets is available at .

OSA Status Determination

The AHI, a gold standard measure of OSA severity,14 and the Epworth Sleepiness Score (ESS, a measure of daytime sleepiness32) were derived from both sleep databases to define the OSA status.

Given the unclear clinical significance of mild OSA,33 we utilized two definitions for clinically significant OSA separately as primary: 1) moderate/severe OSA (Yes/No) defined by AHI ≥15 and 2) severe OSA (Yes/No) defined by AHI >30.14 For the secondary analyses, we utilized two alternative definitions for OSA: 1) AHI ≥5 regardless of the presence of daytime sleepiness and 2) AHI ≥5 and presence of moderate to severe daytime sleepiness as measured by the ESS ≥13 of 24 (). We focused on individuals with moderate to severe daytime sleepiness, given that current evidence is insufficient to assess the balance of benefits and harms of identifying and treating individuals with mild OSA associated with mild symptoms, asymptomatic, or unrecognized symptoms.10

Predictive Model Covariates from Health Administrative Data

presents a health care pattern for individuals with suspected OSA in Ontario (Canada). We used a combination of clinical judgment and empirical evidence to select variables from health administrative data. Following variables were derived from 3 years before to 1 year after the index sleep study: 1) physician billing diagnoses for OSA (OHIP); 2) OSA billing physician specialty more frequently associated with a sleep medicine practice such as internal medicine, respirology, neurology, otolaryngology or psychiatry (IPDB); 3) OSA billing physician being registered with the ADP (Yes/No); 4) diagnostic and/or therapeutic sleep studies (OHIP); 5) inpatient hospitalizations for OSA (DAD/SDS); 6) surgical procedures for OSA such as maxillomandibular advancement surgery, uvulopalatopharyngoplasty, tonsillectomy and/or adenoidectomy or bariatric surgery (DAD/SDS/OHIP); 7) PAP treatment claims (ADP) as recommended treatment for moderate to severe OSA in adults15; and 8) having substance abuse disorder34 (DAD). Finally, we used  9) patient demographics, such as age, sex, income status and place of residence (RPDB/Census) at the year of the sleep study; and 10) prevalent comorbidities at the year of the sleep study and hospitalizations for selected comorbidities in the last 3 years prior to the sleep study, including the Charlson Comorbidity Index35 (ICES-derived databases/OMHRS/OHIP/DAD). Comorbidities were included due to association with OSA,36 and to address an increased exposure to the health care system that may increase the likelihood of a documented OSA diagnosis. Detailed definitions of 39 candidate variables considered are presented in the ).

Analyses

Study results were reported using the Standards for Reporting of Diagnostic Accuracy criteria for assessing the quality of validation studies of health administrative data.37

Descriptive statistics were used to characterize the study populations. Each selected variable was cross-tabulated with the OSA status: sensitivity, specificity, and 95% confidence intervals (CI) of each variable were evaluated.

Development of Case-Ascertainment Model

First, we selected candidate variables (see details above). Then, we screened the list of candidate variables and excluded those with a prevalence of ≤0.5% in the sleep cohort used for the internal validation38 (31 of 39 candidate variables left). Next, we used the classification and regression tree (CART) methods to identify the most parsimonious model with the best predictive characteristics.39 For each OSA status definition, the full tree was pruned back to a more parsimonious tree via cost-complexity pruning to prevent overfitting and select a parsimonious set of variables. The pruning parameter selected was based on the tree with the number of leaves that minimized the average squared error across 10-fold cross-validation.39 Finally, we used the CART and logistic regression techniques to test the internal and external validity of prediction models based on selected variables. Two types of models were tested: 1) parsimonious models selected via cost-complexity pruning (both CART and logistic regressions) and 2) full models (with all 31 selected variables included in logistic regressions only). All individuals with an estimated probability of 0.5 or greater using the case-ascertainment models were classified as having OSA; a default cut point of 0.5 was chosen given the lack of recommendations on the best probability threshold for OSA validation.

Measures of the Predictive Model Performance

C-index, sensitivities, specificities, positive (+LR) and negative likelihood ratios (−LR), and positive (PPV) and negative predictive values (NPV) were used to assess the performance of the case-ascertainment models.40,41 C-index values of 0.70–0.79, 0.80–0.89, and >0.90 were interpreted as having moderate, good, and excellent discriminative ability, respectively.42 Based on the values, LRs as measures of diagnostic accuracy were considered as small (+LR: 2–5; −LR: 0.2–0.5), moderate (+LR: 5–10; −LR: 0.1–0.2) and important (+LR: >10; −LR: <0.1).43

Internal Validation

Optimism corrected methods via the bootstrap were used to internally validate model discrimination.44 The optimism was estimated as the decrease in the c-index when models derived in 1 of 1000 bootstrap samples were applied to the original sample. This optimism was then subtracted from the original estimate to obtain an “optimism-corrected” estimate.45

External Validation

External validity was assessed by the following measures: calibration-in-the-large or the model intercept; calibration slope; discrimination (c-index); and a mean integrated calibration index (ICI).46,47 Calibration plots were used to visualize the agreement between the observed and predicted values; a locally weighted least squares regression smoothing technique (ie, the loess algorithm) was utilized.48

Secondary Analyses

We used the analytic approaches described above for the alternative definitions of OSA. We also refitted our parsimonious models among 1) men and women separately; 2) individuals aged 30 and 69 years, given that OSA prevalence is most often reported in this population3; 3) individuals with prevalent hypertension; 4) individuals who were diagnosed with OSA by a physician registered with the ADP; and 5) among those who claimed a PAP device through the ADP following a diagnostic sleep study. Finally, we tested multiple OSA probability thresholds for the best predictive model identified to be tested in future research.

All statistical analyses were performed in the secure environment at ICES Ottawa following Ontario privacy standards using SAS Enterprise Guide 7.15 (SAS Institute Inc., Cary, NC).

Results

Characteristics of clinical cohorts are presented in Table 1.

Table 1

Characteristics of Adults Who Underwent a Diagnostic Sleep Study at the Ottawa Hospital (TOH) Sleep Center (Internal Validation Cohort) and at the London Health Sciences Centre (External Validation Cohort)

Variable	Internal Validation Cohort (2015–2017)	External Validation Cohort (2007–2015)
	N=5099	N=13,487
Demographics at the date of the index sleep study
Male, n (%)	2719 (53.3)	8047 (59.7)
Age, years, median (IQR)	50 (39–60)	51 (41–60)
Rural location	360 (7.1)	2237 (16.6)
Neighbourhood Income Quintile (Q)
Q1	715 (14.0)	2321 (17.2)
Q2	935 (18.3)	2634 (19.5)
Q3	978 (19.2)	2692 (20.0)
Q4	1281 (25.1)	2822 (20.9)
Q5	1171 (23.0)	2954 (21.9)
Clinical characteristics at the date of the index sleep study
AHI, total, events/hour, median (IQR)	8.6 (2.9–18.9)	12.3 (4.2–30.0)
BMI, kg/m2, median (IQR)	28.9 (25.4–33.5)	32.0 (27.0–37.0)
ESS, median (IQR)	8.0 (5.0–12.0)	9.0 (6.0–13.0)
Prevalent comorbidities at the date of the index sleep study, n (%)
Adenoid or/and tonsillectomy	133 (2.6)	267 (2.0)
Asthma	1120 (22.0)	2277 (16.9)
CHF	194 (3.8)	664 (4.9)
COPD	546 (10.7)	1670 (12.4)
Diabetes	774 (15.2)	2523 (18.7)
Hypertension	1616 (31.7)	5739 (42.6)
Myocardial Infarction	117 (2.3)	402 (3.0)
Events within the 3 years prior to the index sleep study, n (%)
Inpatients and outpatients’ codes for OSA diagnoses	1984 (38.9)	6180 (45.8)
Outpatient OSA visit with a physician registered with ADP	944 (18.5)	4231 (31.4)
Outpatient OSA visit with a specialist physician*	728 (14.3)	2723 (20.2)
Surgical procedures for OSA#	30 (0.6)	85 (0.6)
PAP treatment	35 (0.7)	169 (1.3)
A prior sleep study	171 (3.4)	727 (5.4)
COPD hospitalization	73 (1.4)	323 (2.4)
Depression (hospitalization or outpatient visit)	2154 (42.2)	5387 (39.9)
Arrhythmia hospitalization	125 (2.5)	372 (2.8)
Cancer hospitalization	82 (1.6)	210 (1.6)
Cardiovascular disease hospitalization	198 (3.9)	692 (5.1)
Charlson comorbidity index	498 (9.8)	1758 (13.0)
Diabetes hospitalization	258 (5.1)	948 (7.0)
Hypertension hospitalization	222 (4.4)	908 (6.7)
Obesity hospitalization	63 (1.2)	290 (2.2)
Events within 1 year since the index sleep study
Inpatients and outpatients’ codes for OSA diagnoses	2871 (56.3)	13,414 (99.5)
Outpatient OSA visit with a specialist physician*	2543 (49.9)	11,970 (88.8)
PAP treatment	1052 (20.6)	3681 (27.3)
Surgical procedures for OSA#	52 (1.0)	131 (1.0)
A repeated sleep study	1225 (24.0)	3225 (23.9)
Outpatient OSA visit with a physician registered with ADP	2576 (50.5)	13,405 (99.4)

Notes: *An OSA relevant specialty (OSA billing physician specialty more frequently associated with a sleep medicine scope of practice): internal medicine, respirology, neurology, otolaryngology or psychiatry. #Surgical procedures for OSA: maxillomandibular advancement surgery, uvulopalato-pharyngoplasty, tonsillectomy and/or adenoidectomy or bariatric surgery. Percentages are col percentages may not sum to 100% due to missing values (<0.5%)

Abbreviations: CHF, chronic heart failure; COPD, chronic obstructive pulmonary disease; IQR, interquartile range; Q, quintile.

The Derivation and Internal Validation Cohort

Of 5155 adults who underwent a diagnostic sleep study from 2015 to 2017 at the Ottawa Hospital Sleep center, 5099 (99%) were successfully linked to health administrative data. Patients were middle-aged (median age of 50 years), equally divided between males and females (53% males) and had a median AHI of 8.6, with 3266 (64%) individuals having mild to severe OSA (AHI ≥5), 1664 (33%) having moderate to severe OSA (AHI ≥15) and 643 (13%) having severe OSA (AHI >30). Distributions of variables by the reference standard (primary and secondary definitions) are presented in .

The External Validation Cohort

Of 13,787 adults who underwent a diagnostic sleep study from 2007 to 2015 at the London Health Sciences Centre, 13,486 (98%) were successfully linked to health administrative data. Similar to the derivation cohort, patients were middle-aged (median age of 51 years), although more likely to be males (60% males), and had a higher median AHI of 12.3, with 9687 (72%) individuals having mild to severe OSA, 6006 (45%) having moderate to severe OSA and 3369 (25%) having severe OSA. Distributions of variables by the reference standard (primary and secondary definitions) are presented in .

Sensitivities and Specificities for Individual Variables Considered in Case-Ascertainment Models

In the derivation cohort, the sensitivities of the variables considered in the case-ascertainment models to identify moderate to severe OSA ranged between 0.42% and 83.8%. The variable with the highest sensitivity was an inpatient or outpatient visit for OSA within 1 year since the index sleep study. The specificities ranged between 52.4% and 99.2%. The variable with the highest specificity was a prior surgical treatment for OSA (Table 2). Similar results were noted in the cohort used for the external validation (Table 2), and for individual variables tested to identify severe OSA (Table 3).

Table 2

Sensitivities and Specificities, %, (with 95% Confidence Intervals) for Individual Variables to Be Considered in a Case-Ascertainment Model for Moderate to Severe Obstructive Sleep Apnea (OSA) (AHI ≥ 15)

Variable	Internal Validation Cohort (2015–2017; N=5099)		External Validation Cohort (2007–2015; N=13,487)
	Sensitivity (95% CI)	Specificity (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)
At the date of the index sleep study
Male	65.14 (62.80–67.44)	52.40 (50.72–54.08)	70.65 (69.48–71.80)	49.14 (48.01–50.28)
Rural location	7.81 (6.57–9.21)	93.30 (92.42–94.12)	16.95 (16.01–17.92)	83.70 (82.85–84.53)
Prevalent comorbidities at the date of the index sleep study
Adenoid or/and tonsillectomy	1.62 (1.07–2.35)	96.91 (96.28–97.47)	1.20 (0.94–1.51)	97.39 (97.01–97.74)
Asthma	18.15 (16.32–20.09)	76.19 (74.73–77.60)	14.75 (13.86–15.67)	81.40 (80.50–82.28)
CHF	6.79 (5.63–8.11)	97.64 (97.08–98.12)	6.79 (6.17–7.46)	96.58 (96.14–96.98)
COPD	12.20 (10.66–13.87)	90.01 (88.96–91.00)	14.57 (13.69–15.49)	89.37 (88.65–90.06)
Diabetes	22.66 (20.66–24.74)	88.44 (87.33–89.49)	23.29 (22.23–24.38)	84.97 (84.14–85.78)
Hypertension	45.25 (42.84–47.68)	74.88 (73.39–76.32)	52.40 (51.13–53.67)	65.35 (64.26–66.43)
Myocardial Infarction	3.85 (2.97–4.89)	98.46 (97.99–98.84)	4.18 (3.69–4.72)	97.98 (97.64–98.29)
Events within the 3 years prior to the index sleep study
Inpatients and outpatients’ codes for OSA diagnoses	42.97 (40.57–45.39)	63.06 (61.42–64.67)	42.79 (41.53–44.05)	51.74 (50.60–52.88)
Outpatient OSA visit with a physician registered with ADP	20.55 (18.63–22.58)	82.47 (81.16–83.73)	28.02 (26.89–29.18)	65.94 (64.85–67.01)
Outpatient OSA visit with a specialist physician*	16.23 (14.49–18.09)	86.67 (85.48–87.79)	16.95 (16.01–17.92)	77.21 (76.24–78.15)
Surgical procedures for OSA#	0.42 (0.17–0.86)	99.33 (99.00–99.58)	0.57 (0.39–0.79)	99.32 (99.10–99.49)
PAP treatment	0.66 (0.33–1.18)	99.30 (98.96–99.55)	1.03 (0.79–1.32)	98.57 (98.27–98.83)
A prior sleep study	3.19 (2.39–4.15)	96.56 (95.90–97.15)	3.48 (3.03–3.97)	93.07 (92.48–93.64)
Depression (hospitalization or outpatient visit)	36.06 (33.75–38.42)	54.76 (53.08–56.43)	35.58 (34.37–36.81)	56.55 (55.42–57.68)
COPD hospitalization	1.44 (0.93–2.14)	98.57 (98.12–98.94)	2.86 (2.46–3.32)	97.98 (97.64–98.29)
Arrhythmia hospitalization	3.97 (3.08–5.02)	98.28 (97.79–98.69)	3.95 (3.47–4.47)	98.20 (97.87–98.48)
Cancer hospitalization	2.46 (1.77–3.33)	98.81 (98.38–99.14)	1.90 (1.57–2.28)	98.72 (98.43–98.96)
Cardiovascular disease hospitalization	6.31 (5.19–7.59)	97.29 (96.69–97.81)	7.29 (6.65–7.98)	96.60 (96.17–97.00)
Charlson comorbidity index	14.84 (13.17–16.64)	92.69 (91.77–93.54)	16.88 (15.94–17.85)	90.05 (89.35–90.72)
Diabetes hospitalization	8.59 (7.29–10.04)	96.65 (95.99–97.23)	9.31 (8.58–10.07)	94.80 (94.27–95.29)
Hypertension hospitalization	7.57 (6.35–8.95)	97.21 (96.60–97.73)	9.04 (8.33–9.79)	95.12 (94.61–95.60)
Obesity hospitalization	1.44 (0.93–2.14)	98.86 (98.45–99.19)	2.85 (2.44–3.30)	98.41 (98.10–98.68)
Events within 1 year since the index sleep study
Inpatients and outpatients’ codes for OSA diagnoses	83.83 (81.98–85.57)	57.03 (55.36–58.69)	99.65 (99.47–99.78)	0.68 (0.51–0.90)
Outpatient OSA visit with a specialist physician*	80.59 (78.61–82.46)	65.01 (63.39–66.60)	92.12 (91.41–92.79%)	13.94 (13.17–14.75)
Outpatient OSA visit with a physician registered with ADP	80.71 (78.73–82.58)	64.10 (62.47–65.71)	99.60 (99.41–99.74)	0.76 (0.58–0.99)
PAP treatment	46.21 (43.80–48.64)	91.76 (90.79–92.66)	49.77 (48.49–51.04)	90.75 (90.07–91.40)
Surgical procedures for OSA#	1.50 (0.97–2.21)	99.21 (98.86–99.48)	1.02 (0.78–1.30)	99.06 (98.82–99.27)
A repeated sleep study	52.76 (50.33–55.19)	89.90 (88.84–90.89)	43.72 (42.46–44.99)	91.99 (91.35–92.60)

Notes: *An OSA relevant specialty (OSA billing physician specialty more frequently associated with a sleep medicine scope of practice): internal medicine, respirology, neurology, otolaryngology or psychiatry. #Surgical procedures for OSA: maxillomandibular advancement surgery, uvulopalato-pharyngoplasty, tonsillectomy and/or adenoidectomy or bariatric surgery

Abbreviations: ADP, assistive device program; AHI, apnea–hypopnea index; CHF, chronic heart failure; CI, confidence interval; COPD, chronic obstructive pulmonary disease; OSA, obstructive sleep apnea.

Table 3

Sensitivities and Specificities, %, (with 95% Confidence Intervals) for Individual Variables to Be Considered in a Case-Ascertainment Model for Severe OSA (AHI >30)

Variable	Internal Validation Cohort (2015–2017; N=5099)		External Validation Cohort (2007–2015; N=13,487)
	Sensitivity (95% CI)	Specificity (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)
At the date of the index sleep study
Male	70.30 (66.60–73.81)	49.12 (47.65–50.60)	74.41 (72.90–75.88)	45.24 (44.27–46.22)
Rural location	7.78 (5.83–10.12)	93.04 (92.26–93.77)	17.63 (16.36–18.96)	83.76 (83.03–84.47)
Prevalent comorbidities at the date of the index sleep study
Adenoid or/and tonsillectomy	1.09 (0.44–2.23)	97.17 (96.64–97.64)	1.10 (0.77–1.51)	97.73 (97.42–98.01)
Asthma	17.57 (14.71–20.74)	77.40 (76.14–78.62)	14.43 (13.26–15.66)	82.30 (81.54–83.04)
CHF	8.86 (6.78–11.33)	96.93 (96.38–97.41)	8.34 (7.43–9.33)	96.21 (95.82–96.58)
COPD	11.98 (9.57–14.74)	89.47 (88.54–90.36)	15.41 (14.20–16.67)	88.62 (87.99–89.24)
Diabetes	28.77 (25.30–32.44)	86.78 (85.75–87.76)	26.27 (24.79–27.79)	83.81 (83.08–84.52)
Hypertension	50.08 (46.14–54.01)	70.96 (69.60–72.29)	56.43 (54.73–58.11)	62.06 (61.11–63.01)
Myocardial infarction	4.67 (3.17–6.59)	98.05 (97.60–98.43)	4.33 (3.67–5.08)	97.47 (97.14–97.77)
Events within the 3 years prior to the index sleep study
Inpatients and outpatients’ codes for OSA diagnoses	42.61 (38.75–46.54)	61.62 (60.1–63.06)	43.81 (42.13–45.51)	53.50 (52.53–54.48)
Outpatient OSA visit with a physician registered with ADP	20.37 (17.32–23.70)	81.75 (80.59–82.88)	28.55 (27.03–30.11)	67.69 (66.77–68.60)
Outpatient OSA visit with a specialist physician*	16.64 (13.84–19.75)	86.06 (85.01–87.07)	16.89 (15.64–18.20)	78.71 (77.90–79.50)
Surgical procedures for OSA#	0.31 (0.04–1.12)	99.37 (99.09–99.58)	0.33 (0.16–0.58)	99.27 (99.08–99.43)
PAP treatment	0.47 (0.10–1.36)	99.28 (98.99–99.51)	1.04 (0.72–1.44)	98.68 (98.43–98.89)
A prior sleep study	2.18 (1.20–3.63)	96.48 (95.89–97.00)	2.82 (2.29–3.44)	93.75 (93.26–94.22)
Depression (hospitalization or outpatient visit)	34.99 (31.30–38.82)	56.71 (55.24–58.17)	34.37 (32.77–36.00)	58.20 (57.23–59.16)
COPD hospitalization	0.93 (0.34–2.02)	98.50 (98.09–98.83)	3.21 (2.64–3.86)	97.87 (97.57–98.15)
Arrhythmia hospitalization	5.60 (3.95–7.67)	98.00 (97.55–98.39)	4.66 (3.97–5.43)	97.87 (97.57–98.15)
Cancer hospitalization	3.11 (1.91–4.76)	98.61 (98.22–98.93)	2.26 (1.78–2.82)	98.68 (98.43–98.89)
Cardiovascular disease hospitalization	7.78 (5.83–10.12)	96.68 (96.11–97.19)	8.70 (7.77–9.70)	96.06 (95.66–96.43)
Charlson comorbidity index	17.42 (14.56–20.58)	91.34 (90.47–92.15)	19.47 (18.15–20.85)	89.11 (88.48–89.71)
Diabetes hospitalization	10.42 (8.17–13.04)	95.71 (95.08–96.29)	11.16 (10.12–12.27)	94.35 (93.88–94.79)
Hypertension hospitalization	7.78 (5.83–10.12)	96.14 (95.53–96.69)	10.63 (9.61–11.72)	94.56 (94.10–95.00)
Obesity hospitalization	1.24 (0.54–2.44)	98.77 (98.40–99.07)	3.47 (2.88–4.15)	98.29 (98.02–98.53)
Events within 1 year since the index sleep study
Inpatients and outpatients’ codes for OSA diagnoses	86.78 (83.92–89.30)	48.09 (46.62–49.57)	99.76 (99.53–99.90)	0.63 (0.49–0.81)
Outpatient OSA visit with a physician registered with ADP	84.14 (81.08–86.88)	54.33 (52.86–55.80)	99.76 (99.53–99.90)	0.72 (0.57–0.91)
Outpatient OSA visit with a specialist physician*	83.67 (80.58–86.45)	55.00 (53.53–56.47)	93.65 (92.77–94.45)	12.87 (12.22–13.54)
PAP treatment	53.97 (50.02–57.87)	84.18 (83.07–85.24)	62.39 (60.73–64.03)	84.39 (83.67–85.09)
Surgical procedures for OSA#	1.71 (0.86–3.04)	99.08 (98.75–99.34)	1.01 (0.70–1.41)	99.04 (98.83–99.22)
A repeated sleep study	62.05 (58.18–65.82)	81.46 (80.29–82.59)	48.50 (46.80–50.20)	84.27 (83.55–84.98)

Notes: *An OSA relevant specialty (OSA billing physician specialty more frequently associated with a sleep medicine scope of practice): internal medicine, respirology, neurology, otolaryngology or psychiatry. #Surgical procedures for OSA: maxillomandibular advancement surgery, uvulopalato-pharyngoplasty, tonsillectomy and/or adenoidectomy or bariatric surgery

Abbreviations: ADP, assistive device program; AHI, apnea–hypopnea index; CHF, chronic heart failure; CI, confidence interval; COPD, chronic obstructive pulmonary disease; OSA, obstructive sleep apnea.

Performance of a Case-Ascertainment Model: Model Derivation and Internal Validation

Table 4 represents performance measures for both CART and regression models to identify individuals with primary OSA case definitions. Details on variables selected, including importance, regression trees, and node information for parsimonious models, are presented in the Data Supplements (). The parsimonious CART model to identify individuals with moderate to severe OSA contained six variables in relation to an index sleep study: an outpatient visit for OSA from a specialist physician, a repeated sleep study and a PAP treatment claim within 1 year of the index sleep study, patient sex and age at the index sleep study, and hospitalization with hypertension in the last 3 years prior to the sleep study (). For example, an individual who underwent a repeated sleep study and claimed a PAP treatment within 1 year of the index sleep study had an 80% probability of having moderate to severe OSA, while an individual who had not had a repeated sleep study and an outpatient visit for OSA from a specialist physician within 1 year of the index sleep study had a 10% probability of having OSA (). The parsimonious CART model for severe OSA also contained six variables: an outpatient visit for OSA from a physician registered with the ADP to prescribed PAP therapy, a repeated sleep study and a PAP treatment claim within 1 year of the index sleep study, patient age and prevalent diabetes at the index sleep study, and hospitalizations for cardiovascular comorbidities in the last 3 years prior to the sleep study (). For example, an individual with diabetes and older than 60 years at the index date, who underwent a repeated sleep study and had an outpatient visit for OSA from a physician registered with the ADP within 1 year since the index date, had a 78% probability of severe OSA. In contrast, an individual who had not had a repeated sleep study within 1 year since the index date, had a 6% probability of severe OSA ().

Table 4

Internal and External Validation Measures of CARTs and Logistic Regression Models Developed to Identify Individuals with Moderate to Severe or Severe Obstructive Sleep Apnea (OSA)

	c-Statistic*		Sensitivity#		Specificity#		Positive Predictive Value		Negative Predictive Value		+LR#		−LR#
	Int	Ext	Int	Extern	Int	Ext	Int	Ext	Int	Ext	Int	Ext	Int	Ext
Apnea–hypopnea index (AHI) ≥15: Moderate to severe OSA
Classification regression tree (CART) Models
Parsimonious¶	0.75	0.75	0.66 (0.63–0.68)	0.60 (0.59–0.61)	0.87 (0.86–0.88)	0.88 (0.87–0.88)	0.71 (0.68–0.73)	0.79 (0.78–0.81)	0.84 (0.83–0.85)	0.73 (0.72–0.74)	5.1	5.0	0.4	0.5
Logistic Regression Models
Parsimonious¶	0.84	0.81	0.61 (0.59–0.63)	0.59 (0.58–0.60)	0.89 (0.87–0.90)	0.87 (0.87–0.88)	0.72 (0.70–0.74)	0.79 (0.78–0.80)	0.82 (0.81–0.84)	0.73 (0.72–0.74)	5.6	4.5	0.4	0.5
Full^	0.85	0.81	0.61 (0.59–0.63)	0.62 (0.61–0.63)	0.89 (0.88–0.90)	0.85 (0.85–0.86)	0.72 (0.70–0.75)	0.77 (0.76–0.78)	0.82 (0.81–0.84)	0.74 (0.73–0.75)	5.6	4.1	0.4	0.5
AHI >30: Severe OSA
CART Models
Parsimonious¶	0.66	0.67	0.14 (0.11–0.17)	0.07 (0.06–0.08)	0.99 (0.98–0.99)	0.99 (0.98–0.99)	0.61 (0.53–0.69)	0.62 (0.57–0.67)	0.89 (0.88–0.90)	0.76 (0.75–0.77)	14.0	7.0	0.9	0.9
Logistic Regression Models
Parsimonious¶	0.80	0.80	0.13 (0.11–0.16)	0.10 (0.09–0.11)	0.98 (0.98–0.99)	0.98 (0.98–0.98)	0.53 (0.45–0.61)	0.62 (0.57–0.66)	0.89 (0.88–0.90)	0.77 (0.76–0.77)	6.5	5.0	0.9	0.9
Full^	0.82	0.81	0.16 (0.13–0.19)	0.13 (0.12–0.15)	0.98 (0.98–0.98)	0.98 (0.97–0.98)	0.55 (0.47–0.62)	0.65 (0.61–0.68)	0.89 (0.88–0.90)	0.77 (0.76–0.78)	8.0	6.5	0.9	0.9

Notes: *Bootstrap Optimism Corrected for the internal validation (rep=1000). #Sensitivity, specificity, and positive and negative likelihood ratios were calculated as based on an estimated probability of 0.5 or greater to classify individuals as having OSA. ¶Parsimonious models: AHI ≥15: an outpatient visit for OSA from a specialist physician, a repeated sleep study and a PAP treatment claim within 1 year of the index sleep study, patient sex and age at the index sleep study and hospitalizations with hypertension in the last 3 years prior to the sleep study. AHI >30: an outpatient visit for OSA from a physician registered with the ADP, a repeated sleep study and a PAP treatment claim within 1 year since the index sleep study, patient age and prevalent diabetes at the date of a sleep study, hospitalizations for cardiovascular comorbidities in the last 3 years prior to the sleep study. ^Details on variables included in parsimonious and full models are presented in the Data Supplement.

Abbreviations: +LR, positive likelihood ratio; −LR, negative likelihood ratio.

Estimates from logistic models utilizing variables derived through the CART method and from full logistic regression models are presented in the Data Supplement ().

All CART and logistic models demonstrated moderate to good discriminative ability to identify individuals with moderate to severe OSA (optimism corrected c-indices of 0.75–0.85). The sensitivities ranged from 61% to 66%, specificities from 87% to 89%, +LRs from 5.1 to 5.6, −LRs were 0.4, PPV from 71% to 72% and NPV from 83% to 84% (Table 4).

Logistic regression models demonstrated good discriminative ability to identify individuals with severe OSA (optimism corrected c-indices of 0.80–0.82). The sensitivities ranged from 13% to 16%, specificities from 98% to 99%, +LRs from 6.5 to 14.0, −LRs were 0.9, PPV from 53% to 61% and NPVs were 89% (Table 4).

Performance of a Case-Ascertainment Model: External Validation

represents all calibration indices. Case-ascertainment models performed better to identify moderate to severe than severe OSA (mean integrated calibration indices of 0.02–0.06 and 0.08–0.12, respectively) (Figures 1 and 2). Logistic models demonstrated better model performance and discriminative ability than CART models. All logistic models had good discriminatory ability for primary OSA definitions (c-indices of 0.80–0.81; Figure 2; Table 4). Compared to the initial models, for moderate to severe OSA, and for all types of models, in general, the sensitivity decreased to approximately 60% and the specificity remained at 95% and above. We found an improvement in PPV up to 79% and a drop in NPV to 73% with +LRs of 4.1–5.0 and –LRs of 0.5 (Table 4). Compared to the initial models for severe OSA, the sensitivity decreased to 7%, while the specificity remained high at 98–99%. We found an improvement in PPV up to 65% and a drop in NPV to 76% with +LRs of 5.0–7.0 and –LRs of 0.9, suggesting moderate diagnostic accuracy for ruling-in severe OSA diagnosis, but poor ability to rule-out the diagnosis. Both parsimonious and full logistic regression models demonstrated similar measurement properties, suggesting no need to consider all variables in the predictive model.

Figure 1

External validation: Calibration plots to classify individuals with moderate to severe (apnea-hypopnea index [AHI] ≥15) or severe (AHI >30) obstructive sleep apnea (OSA) for the parsimonious classification and regression tree (CART) and logistic regression models (same variables), and full logistic regression models. Perfect predictions should be on the ideal diagonal line, described with an intercept of 0 and slope of 1. Imperfect calibration can be characterized by deviations from these ideal values.

Figure 2

External validation: Area under the curve to classify individuals with moderate to severe obstructive sleep apnea (OSA) (apnea-hypopnea index [AHI] ≥15) utilizing parsimonious classification and regression tree (CART) and logistic regression models (same variables).

Alternative Definitions

Measurement characteristics for individual variables and details on parsimonious and full models to identify individuals with OSA utilizing alternative definitions are presented in the Data Supplement ().

On the internal validation cohort, all models demonstrated moderate to good discriminatory ability to identify mild to severe OSA regardless of daytime sleepiness (optimism corrected c-indices of 0.77 to 0.84) (). In general, all models demonstrated similar sensitivities of 81–82%, specificities of 66–70%, PPVs of 81–83%, NPVs of 67–68%, +LRs of 2.4–2.7, and −LRs of 0.27 demonstrating small diagnostic accuracy. On the external validation cohort, all models exhibited higher sensitivities of 95–97%, lower specificities of 19–25%, PPVs of 75–76%, NPVs of 65–71%, +LR of 1.2–1.3, and −LR of 0.16–0.20 demonstrating moderate diagnostic accuracy in ruling-out the OSA diagnosis ().

We were not able to identify a parsimonious model to define mild to severe OSA cases with the presence of moderate to severe daytime sleepiness. The full logistic model with all 32 variables considered did not demonstrate reasonable discriminative or diagnostic ability ().

Subgroup Analysis

Refitting all case-ascertainment models on subgroups demonstrated similar patterns to the entire sample among females, males, individuals aged between 30 and 69 years, with prevalent hypertension, and individuals seen by a physician registered with the ADP ().

Exploring Different Probability Thresholds

presents the sensitivities, specificities, PPVs, NPVs, +LRs and −LRs for the different probability thresholds for the parsimonious logistic models in the external validation cohort. The largest increase in the post-test probability of moderate to severe OSA was noted for the threshold of about 90% (+LRs of 14–49), with the largest decrease in the post-test probability of the disease for the thresholds of 5% (−LR of 0.08). For severe OSA, the largest increase in the post-test probability of the disease was noted for the threshold of 66% (+LR of 7), with the largest decrease in the post-test probability of the disease for the thresholds of 2% (−LR of 0.12).

Discussion

In our retrospective cohort study of adults referred for sleep assessment (and therefore with a higher pre-test probability of having OSA than the general population), we derived and validated administrative data case-ascertainment models to identify individuals with OSA. When individuals were classified as moderate to severe OSA (model-based OSA probability exceeded 0.50), our case-ascertainment models demonstrated relatively low sensitivity, high specificity and good discriminative ability, with reasonably high PPV and NPV values. Case-ascertainment models for severe OSA demonstrated higher specificity, but had even lower sensitivity; these models had a moderate diagnostic accuracy for ruling-in severe OSA, but poor ability to rule-out a diagnosis of severe OSA. This suggests that these approaches could be used to support future research where identifying a cohort with a high probability of truly having OSA is required, such as evaluating trends and outcomes of individuals with clinically significant OSA using routinely collected health care data on the population level. However, because of misclassification risk, such approaches would not be reliable for comparing the outcomes between those classified and not classified as having severe OSA.

The case-ascertainment models developed and validated in our study demonstrated better performance than the case-ascertainment algorithms from previous studies utilizing health administrative data,24,25 confirming our hypothesis, and previous research,49 that multivariable modelling techniques can improve the accuracy of case-ascertainment algorithms. In contrast to our study, previous studies did not employ internal and external validations, which suggests that their findings may have been overly optimistic. Notably, and consistent with a previous study performed in the same jurisdiction,24 we demonstrated the importance of a repeated sleep study and a PAP claim in identifying individuals with OSA in Ontario. Further, we confirmed that inpatient and outpatient codes for OSA diagnoses cannot be utilized on their own to accurately identify OSA.16 This is likely due to several factors, including poor specificity of coding choices available in acute care discharge abstracts and physician billing records (likely due to inaccurate coding by the doctors and hospitals involved),16 as well as differences in practice and billing between institutions. As an example, the Ontario OHIP diagnostic codes for OSA, 307 and 786, represent not only OSA but also tension headaches, anorexia nervosa, or shortness of breath.16 Our study also highlights center-level variation in diagnostic coding that can lead to misclassification. For example, in one of our study centers, almost all individuals who underwent a PSG independent of the resulting AHI were assigned inpatient or outpatient codes for OSA diagnoses, whereas our other center assigned OSA diagnostic codes to only about half of individuals. This suggests that in some centers, diagnostic codes may be used to mark the indication for a PSG, whereas in others, the code may be assigned instead based on the PSG results. Finally, the cohort used for external validation was somewhat different from the derivation cohort (higher prevalence of male sex, obesity, hypertension, and severe OSA). The two cohorts were from two different sleep clinics and represented real-world variation between sleep clinics’ populations. As such, good model performance on the external cohort suggests robustness of the developed models. This again stresses the importance of multivariable models and external validation.

When comparing our case-ascertainment models to existing OSA clinical screening tools, we found, as expected, that screening tools were characterized by a higher sensitivity and lower specificity. A systematic review of eight patient-based questionnaires for OSA screening revealed a pooled sensitivity for predicting moderate OSA of 77% (73–81%) and a pooled specificity of 44% (41–47%).50 The PPV and NPV ranged from 11% to 97% and from 48% to 97%, respectively, with the prevalence of moderate to severe OSA between 8% and 70%.50 The pooled + LRs ranged from 2.0 to 2.8, and – LRs from 0.2 to 0.6.51 Importantly, test accuracy in repeated validation studies of the same screening tool was variable, suggesting an underlying heterogeneity in either measured elements of these predictive models or the clinical presentation of OSA.51 The non-specific clinical presentation of OSA requires the use of diagnostic tests among many individuals who do not in fact have OSA to identify cases. This results in a reduced specificity for administrative data codes. High heterogeneity in OSA presentation was also likely the reason our model performed poorly while incorporating measure of daytime sleepiness and highlights the challenge of using health administrative data to identify OSA clinical subtypes. To summarize, while clinical screening tools are potentially valuable to use on an individual level – especially in a certain clinical context and among specific subgroups – our ascertainment models are meant to be used on health administrative databases, rather than on an individual level, to identify a high probability OSA cohort. However, in general, to do meaningful OSA research, we need better diagnostic testing and documentation of these results.

Our study did not find that CARTs were advantageous over conventional logistic regression models for prediction.52,53 This may be because logistic regression performs better with a small sample size, and when non-linear and non-additive signals are not strong enough to make modern machine learning methods advantageous.54

The generalizability of the proposed case-ascertainment models was limited due to a higher pre-test probability of OSA in this population. Future studies should validate our models among individuals not referred for a sleep assessment, and in other jurisdictions with different clinical and billing practices. Next, moderate diagnostic accuracy demonstrated in our study can be partly explained by the main analyses being based on a 0.5 OSA probability threshold. The default of choosing p>0.5 is usually far from the best choice. Thus, additional research is warranted to test different probability thresholds for the predictive models to classify individuals with OSA and discriminate between different levels of disease severity. Finally, we were not able to improve PPV despite utilizing multivariable models, alternative definitions, and populations, suggesting that a better coding system in health administrative data for OSA may be necessary.

Conclusion

Among adults who were referred for a sleep disorder assessment and underwent a diagnostic sleep study in Ontario, case-ascertainment models derived from health administrative data for identifying moderate to severe OSA had relatively low sensitivity but high specificity and good discriminative ability. These findings could help to study trends and outcomes of individuals with OSA using routinely collected health care data. Future studies are required to validate these models among individuals not referred for a sleep assessment, to discriminate between different levels of disease severity, and to test different probability thresholds for the case-ascertainment models to classify individuals with OSA.