Application of drug-induced sleep endoscopy in patients treated with upper airway stimulation therapy.

OBJECTIVE: To determine the level of agreement among experienced operators of candidacy for upper airway stimulation (UAS) based on evaluation of drug-induced sleep endoscopy (DISE).
METHODS: The trial was designed as a single-blinded cross-sectional study. Four otolaryngologists with extensive DISE experience were given 63 video clips from the STAR trial video library. These videos were graded using the VOTE classification. Percentage agreement and Cohen's κ (for inter-rater reliability) were calculated between pairs of reviewers, assessing palatal complete concentric collapse (CCC) and determining UAS eligibility. Subjects were also grouped based on collapse severity for each reviewer.
RESULTS: The reviewers had excellent (approximately 90%) agreement on findings at the level of the soft palate and tongue base. The inter-rater reliability for palatal CCC ranged from moderate to substantial. The agreement on determining the criteria for UAS implantation ranged from poor to moderate. All 4 upper airway structures as classified by the criteria of the VOTE were graded by all the reviewers as contributing to obstruction in a majority of subjects who were performed via application of DISE.
CONCLUSION: Application of DISE remains a subjective examination, even among those experienced operators, therefore more studies need to be performed for evaluation of improvement in inter-rater reliability after implantation of training videos.

Introduction

As an implantable hypoglossal nerve stimulation system, upper airway stimulation (UAS), was recently developed to treat patients with moderate-to-severe obstructive sleep apnea (OSA) who do not tolerate continuous positive airway pressure (CPAP). The effectiveness of UAS was demonstrated in the STAR trial, a prospective, multi-institutional clinical trial. The subjects were screened with drug-induced sleep endoscopy (DISE) in order to exclude those patients with complete concentric collapse (CCC) of the velopharynx (soft palate) because previous studies showed that the patients in this group did not have adequate reductions in their apnea hypopnea index (AHI) following implantation of UAS. Thus, identification of palatal CCC was an important exclusion criterion to determine the eligibility for UAS implantation.

DISE was developed to visualize upper airway obstruction in sedated patients during simulated sleep. Since palatal CCC was associated with poor outcomes for UAS implantation, it was performed in all the subjects who potentially qualified for UAS prior to implantation. Assessment of the pattern of obstruction via DISE is inherently subjective and the reliability ranged from poor to substantial in previous studies.3, 4, 5 The aims of this study were to determine the level of agreement among experienced DISE operators via the VOTE criteria, and scoring to determine if the patient is eligible for UAS implantation.

Methods

Study design

The STAR trial (MUSC IRB HR#20673) is a prospective multi-center clinical trial to determine the safety and efficacy of UAS in CPAP-intolerant adults with moderate-to-severe OSA. Subject eligibility criteria and study design have been published previously. The present study is a single-blinded cross-sectional study to utilize video clips collected as a part of the STAR trial in order to determine the inter-rater reliability among the experienced DISE operators and to assess their agreement in determining the eligibility criteria for UAS implantation. The study protocol was approved via an investigational review board.

DISE

The STAR trial investigators randomly chose DISE video clips in a population of 63 patients who were screened as a part of the STAR trial. No subject data (including demographic data and whether the subjects were ultimately implanted with UAS) were included with the videos. The videos were loaded on a secure website that can be accessed for scoring. Each DISE clip lasted approximately 2 min and the analyses were completed within 3 weeks by the reviewers.

Each DISE video clip was graded using the VOTE scoring system. The responses were compared based on the presence of collapse at the velum, or opharynx, tongue, or epiglottis using dichotomous values (yes or no). Less attention was given to the type of collapse in each area with the exception of the presence or absence of palatal CCC. Each reviewer then determined whether the subject would be a suitable candidate for UAS implantation based on the results collected from application of the DISE. The VOTE score was calculated as the sum of the locations with the obstruction present for a maximum score of 4.

Statistical analysis

All the statistical analyses were performed using SPSS (version 23; IBM, Armonk, New York). Both the percentage of agreement and Cohen's kappa (κ) were used to measure the agreement between two raters. The two raters either agreed in their rating (i.e. the category that a subject is assigned to) or they disagreed; there were no degrees of disagreement (i.e. no weightings). The Cohen's kappa cannot be computed when there was no variation between the two raters. The levels of agreement were not aligned based on the Cohen's kappa. Interpretation of Cohen's κ as introduced by Landis and Koch is presented in Table 1.

Table 1

Cohen's κ interpretation.

Value of κ	Strength of Agreement
<0	No better than chance
0.01–0.20	Poor
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Substantial
0.81–1.00	Almost perfect

Results

A total of 63 DISE videos extracted from the STAR trial video library were reviewed by four experienced DISE operators who were participating investigators in the STAR trial. The percentage of agreement between the pairs of reviewers is presented in Table 2. The reviewers consistently agreed on the presence or absence of obstruction at the level of the soft palate and oropharynx in most cases (approximately 90%) while less agreement can be observed at the tongue base and epiglottis as well as the presence or absence of palatal CCC. The reviewers were not consistently aligned with the criteria for UAS implantation.

Table 2

Percentage of agreement between reviewers (%).

	1&2	1&3	1&4	2&3	2&4	3&4
V	100	87.3	96.8	87.3	96.8	87.3
CCC	65.1	79.4	87.3	76.2	71.4	85.7
OP	100	85.7	95.2	85.7	95.2	81.0
TB	68.3	76.2	87.3	73.0	68.3	85.7
E	71.4	79.4	92.1	76.2	73.0	84.1
UAS	61.9	73.0	76.2	69.8	54.0	71.4

V: velum; CCC: palatal complete concentric collapse; OP: oropharynx; TB: tongue base; E: epiglottis; UAS: upper airway stimulation.

The Cohen's κ was calculated to determine inter-rater reliability (Table 3). The best inter-rater reliability was obtained by assessing palatal CCC and was aligned by one pair of reviewers. The evaluation of the obstruction at the tongue base was more reliable than that at the epiglottis. No consensus was aligned in determining the criteria for UAS implantation with the results that were not better than what would be expected by chance.

Table 3

Cohen κ between reviewers.

	1&2	1&3	1&4	2&3	2&4	3&4
V	N/A∗	N/A∗	N/A∗	N/A∗	N/A∗	Poor
CCC	Poor	Moderate	Substantial	Moderate	Fair	Moderate
OP	N/A∗	N/A∗	N/A∗	N/A∗	N/A∗	Chance
TB	Fair	Fair	Fair	Moderate	Fair	Moderate
E	Poor	Poor	Fair	Fair	Poor	Fair
UAS	Moderate	Poor	Chance	Fair	Chance	Poor

V: velum; CCC: palatal complete concentric collapse; OP: oropharynx; TB: tongue base; E: epiglottis; UAS: upper airway stimulation. *The kappa statistic cannot be computed when one variable is constant.

The collapse severity was graded as the sum of VOTE scores ranging from no collapse (0) to obstruction at all the sites (4). For each reviewer, the number of subjects with each collapse severity is summarized in Table 4.

Table 4

Subjects grouped by number of subsites of collapse.

	Reviewer 1	Reviewer 2	Reviewer 3	Reviewer 4
No collapse	0	0	0	0
Single-level collapse	0	0	4	0
Two-level collapse	2	15	15	7
Three-level collapse	3	13	6	5
Four-level collapse	58	35	38	51

The majority of reviewers categorized the subjects with at least two sites of obstructions via application of the DISE. The obstruction at all the sites (4-level collapse) was the most common collapse severity as measured by all the reviewers. The agreement on the inter-rater reliability for multiple-level collapse as measured by the reviewers ranged from slight to fair (Table 5).

Table 5

Inter-rater reliability for multi-level collapse.

Reviewer Pairs	Kappa Interpretation
1&2	Slight
1&3	Slight
1&4	Fair
2&3	Fair
2&4	Fair
3&4	Fair

Discussion

DISE is a highly subjective examination among medical experts. Although reviewers had excellent agreement for the presence of obstruction at the soft palate and oropharynx, and moderate to substantial agreement identifying palatal CCC, they generally showed poor inter-rater reliability when determining eligibility for UAS therapy. More extensive training on the application of DISE are recommended to obtain more consistent results among the reviewers before UAS implantation.

The experts agreed on the presence of obstruction at the level of the soft palate and oropharynx approximately 90% of the time, with two experts agreeing 100% of the time at both sites. Interestingly, the obstruction at the soft palate appears to benefit from application of UAS due to the physical connections between the palate and the tongue base via the palatoglossus muscle. The imaging studies that assess the upper airway in the treatment with acute UAS confirm the resulting opening of the velopharynx.9, 10 As the reviewers consistently identified obstruction at the oropharynx, especially for the soft palate, they may be more inclined to recommend UAS therapy once tongue base and soft palate obstruction are present and palatal CCC is absent.

The presence of palatal CCC on the DISE may be arguably the most important finding when eligibility for UAS implantation is determined. Although one pair of reviewers showed poor inter-rater reliability, all other pairs demonstrated agreement on the presence or absence of palatal CCC ranging from fair to substantial. A previous study showed that experienced DISE operators have higher observer agreement than non-experienced operators; however, the wide range of agreement in this study may have occurred for a few reasons. A conservative observer may predict that those patients with partial but incomplete concentric collapse would do poorly and therefore should be ineligible for implantation, whereas a more inclusive observer would include all patients without obvious CCC. In addition, there are several recognized patterns of CCC. The most obvious pattern is a funnel-shaped sphincteric collapse, however a more subtle pattern of CCC appears to have the usual AP soft palate collapse until the interaction with collapsible lateral pharyngeal walls is observed with a segment of CCC. The importance of identifying palatal CCC was evident in the early-phase feasibility studies of UAS therapy which showed that patients with palatal CCC had no change in AHI after implantation, and had minimal benefit from UAS therapy; however, only a small subset of patients were studied.1, 11 In the STAR trial, the patients with palatal CCC were excluded and the relatively high rate of surgical success in that trial (67%) appeared to validate the exclusion criteria.

In general, the reviewers had poor agreement in recommending UAS therapy. This findings emphasizes the importance of training in application of DISE to identify specific collapse patterns. A voluntary service is provided by the sponsor (Inspire Medical, Maple Grove, MN) to implanting surgeons that allows them to upload DISE video clips on a server for other experts to review in order to determine whether a patient is a good candidate for implantation. Given the relatively high disagreement among experts in this study, it appears that the patients would benefit from expansion of this voluntary program which may remove the bias of the on-site surgeons who may be inclined to offer the patient a procedure that the patient desperately wants but may ultimately be of little benefit to the patient.

Although the variances exist among the reviewers, a majority of subjects were observed with collapses at all the four sites of the upper airway. These findings corroborate prior studies, which found that multilevel collapse patterns are increasingly prevalent in patients with moderate-to-severe OSA.11, 12 The presence of multilevel collapse demonstrates the difficulties in completely resolving OSA when performing single-level sleep surgery. Application of UAS is a single procedure and improves upper airway obstruction at multiple sites, especially for those sites at the tongue base and soft palate. In addition, it has lower postoperative morbidity compared to traditional multilevel sleep surgeries such as uvulopalatopharyngoplasty (UPPP) and tongue procedures. Although applications of UAS and UPPP have different surgical approaches, the decreased pain levels in the patients who received UAS implantation may be a key factor for deciding the treatment regimens.13, 14, 15 Since application of UAS becomes more popular, the role of traditional multiple-level sleep surgery needs to be reevaluated if the patients are also eligible for UAS therapy.

Several possible explanations exist for the low-level agreement among the observers in the study. Firstly, application of DISE is difficult to objectify and be reduced to individual components that can be effectively graded. Secondly, in order to maximize efficiency of the study, the video clips were limited to 2 min which captured stable respiration as measured by maximal collapsibility. Given the short segments, the observers may have made assumptions of the collapsibility of the airway that may have been observed if the video clips were observed in their entirety. Lastly, the videos varied in quality and some observations may have been missed due to airway secretions, grainy quality, or poor resolution.

There were some other limitations to this study. Despite all the reviewers own extensive experiences on application of DISE, the training videos on UAS implantation were not employed to obtain alignment and consistency in grading upper airway obstruction prior to this study. The reviewers may have graded contribution of collapse differently, and did not decide whether partial collapse was considered “yes” or “no” as presence of obstruction. In addition, the reviewers did not provide reasons why they did not recommend UAS implantation if the palatal CCC was not present, which may suggest that other patterns of obstruction are thought to contribute to efficacy of UAS.

As observed in this study, despite the reviewers own some experiences on application of DISE, the reviewers did not always consistently get aligned with the presence of upper airway obstruction or criteria for implantation of UAS. Those surgeons who lacked of experiences on application of DISE should utilize the Inspire service to determine the criteria for implantation of UAS until a certain comfort level with application of DISE is obtained. The studies could be performed to compare the outcomes of UAS in those patients whose images collected from application of DISE was reviewed by a panel of experts compared to the patients whose images collected from application of DISE was interpreted by the on-site surgeons only.

Conclusion

Application of DISE is a subjective examination among the experienced operators, although there are critical potential limitations. The studies that are designed to evaluate the improvements on the inter-rater reliability after training videos need to be further implemented.

Financial disclosures and conflicts of interest

Dr. Kezirian has the following disclosures: Inspire Medical Systems (Consultant), Nyxoah (Consultant, Equity), ReVENT Medical (Consultant, Equity), Gerard Scientific (Consultant, Equity), Split Rock Scientific (Consultant, Equity), Berendo Scientific (Consultant, intellectual property rights), Magnap (intellectual property rights). Dr. Woodson is a study investigator and consultant for Inspire Medical Systems, a consultant and receives royalty from Medtronic, and consults for Siesta Medical. Dr. de Vries is member of the Medical Advisory Board of NightBalance, consultant of Philips Healthcare and Olympus, researcher for Inspire Medical Systems, member of ReVent's Medical Advisory Board, and has shares in NightBalance and ReVent. Dr. Gillespie has received research support from Inspire Medical Systems and Olympus, and is a consultant for Omniguide, Inspire Medical, and Cook Medical. The other authors have no financial disclosures or conflicts of interest.