Development of a novel screening tool for predicting Cochlear implant candidacy.

OBJECTIVES: Cochlear implantation (CI) is a well-established treatment for sensorineural hearing loss. Due in part to a lack of referral guidelines, CI technology remains underutilized, and many patients who could benefit from CI may not be referred for evaluation. This study aimed to develop a model for predicting CI candidacy using routine audiometric measures, with the goal of providing guidance to clinicians regarding when to refer a patient for CI evaluation.
METHODS: Unaided three-frequency pure tone average (PTA), unaided speech discrimination score (SDS), and best-aided sentence recognition testing with AZBio sentence lists were collected from 252 subjects undergoing CIE. Candidacy was defined by meeting traditional (AZBio score ≤ 60%), or Medicare criteria (≤40%). A logistic regression model was developed to predict candidacy. Confusion matrices were plotted to determine the sensitivity and specificity at various probability thresholds.
RESULTS: Logistic regression models were capable of predicting probability of candidacy for traditional criteria (P < .001) and Medicare criteria (P < .001). PTA and SDS were significant predictors (P < .001). Using a probability cutoff of .5, the models yielded a sensitivity rate of 91% and 78% for traditional and Medicare criteria, respectively.
CONCLUSION: Probability of CI candidacy may be determined using a novel screening tool for referral. This tool supports individualized counseling, serves as a proof of concept for candidacy prediction, and could be modified based on an institution's philosophy regarding an acceptable false positive rate of referral. LEVEL OF EVIDENCE: 4.

INTRODUCTION

Cochlear implantation (CI) is a well‐established and effective therapeutic modality for adults with moderate‐to‐profound sensorineural hearing loss (SNHL). Providers typically use routine audiometric tests such as unaided pure tone audiometry, speech reception thresholds, and word recognition scores to make the decision regarding referral for a formal cochlear implant evaluation (CIE). During a CIE, an audiologic evaluation of speech recognition in the best‐aided condition is performed, with or without the incorporation of background noise. Modern CIE includes a battery of tests including sentence recognition, such as AZBio sentences, and/or monosyllabic word recognition. , In the United States, CI candidacy is ultimately defined by Food and Drug Administration (FDA) labeling for each device manufacturer as well as Medicare or private insurance requirements. Private insurance companies typically require patients to score ≤ 60% on sentence recognition in the best‐aided condition (with or without noise), which is less restrictive than Medicare, which requires scores of ≤40%.

Despite the widely accepted efficacy and cost‐effectiveness of the device, only 5% to 10% of patients who would benefit from CI undergo CIE to formally determine candidacy. , , , , , , , , , , This underutilization of technology is secondary to several factors, including but not limited to limited regional access, poor understanding of the technology by primary care providers resulting in low referral rates for hearing loss, and lack of established referral guidelines for clinicians. , , , , With regards to the question of when to refer a patient, there are currently no established guidelines, and candidacy testing protocols (eg, incorporation of noise or signal‐to‐noise ratio) may vary greatly among institutions. , , , , At the discretion of their provider, patients with moderate‐to‐profound SNHL, poor word recognition, and/or dissatisfaction with hearing aids may be referred for CIE. Although the decision to refer patients with greater degrees of hearing loss (eg, bilateral severe to profound SNHL) and impaired speech recognition may appear straightforward, these patients and especially patients with “borderline” performance on routine audiologic testing (eg, pure tone thresholds, speech discrimination scores) may be overlooked for referral, when in fact they may ultimately benefit from a CI.

Recently, several investigators have sought to evaluate the utility of routine unaided testing parameters that may be used to identify potential CI candidates. , , , , , , , , , In a study of audiometric measurements in a population of 185 participants, four‐frequency pure‐tone average (PTA) and maximum speech understanding for monosyllabic words (PBmax) were found to be suitable metric. By using a probabilistic machine learning model, the authors proposed that if the expression “PBmax [%] < 4FPTA [dB] – 8” was valid for either ear, the patient should be referred for CIE. This screening tool yielded a sensitivity and specificity of 87% and 91%, respectively. In another study, Gubbels et al extrapolated that individuals with lower frequency, pure tone thresholds >75 dB hearing level (HL) and/or monosyllabic word recognition scores <40% have a greater than 80% chance of qualifying for CI based on Medicare criteria. Most recently, Zwolan and colleagues developed a “60/60” rule, which proposes that a PTA ≥60 dB HL and monosyllabic word score ≤60% in the better ear is a reliable predictor that the patient will meet either “traditional” or Medicare requirements. In their cohort of 415 patients, this guideline had a sensitivity of 96% for traditional candidacy. , Moreover, 94% of patients over 65 years of age who met Medicare criteria also met the “60/60” rule. ,

These studies were the first to attempt to use elements from routine unaided audiometry to aid in identifying CI candidates; however, recommendations vary between studies with respect to pure tone thresholds vs PTA, SDS scores, type of word tests used, and protocol for incorporation of background noise. Thus, providers may remain hesitant to refer based on these ranges of audiometric cutoffs or based on a testing protocol that differs from their own. An ideal screening tool for CI referral would be a simple and clinically applicable prediction of CI candidacy probability based on routine audiometric testing. In this study, we aimed to develop a model for CI candidacy prediction based on routine audiometric testing using logistic regression. In contrast to previously described screening strategies, our model intends to provide the specific probability of candidacy for an individual as measured by sentence testing in quiet. Predicting the probability of candidacy potentially allows for adjustment of referral thresholds, more informed decision‐making, and individualized counseling with respect to the decision to pursue cochlear implant evaluation.

MATERIALS AND METHODS

Participants

After obtaining ethics approval from the Ohio State University institutional review board, a retrospective chart review identified a total of 962 adult patients over the age of 18 years who were referred for CIE at a tertiary care center between July 1989 and May 2020. Patients with retrocochlear pathology, single‐sided deafness, previous CI, known cognitive impairment, and patients with incomplete data available for unaided pure tone thresholds, unaided speech discrimination scores, and aided AzBio scores in quiet were also excluded, yielding 335 subjects, including 144 considered Medicare beneficiaries (ie, age ≥ 65 years). From this dataset, 252 subjects demonstrated air conduction pure‐tone thresholds indicative of at least moderate SNHL (ie, ≥40 dB HL at 250, 500, 1000, 2000, 3000, 4000, 6000, 8000 Hz, bilaterally). Eighty‐six (60%) of those ≥65 years old demonstrated at least one pure‐tone threshold in the profound range (≥90 dB HL) bilaterally and were included in the group of Medicare beneficiaries based on strict adherence to Medicare PTA criteria for CI candidacy. Demographic and audiologic data are summarized in Table 1. A total of 12 included cases were those of patients who were referred to the clinic more than once. These evaluations occurred from 1 to 7 years apart (M = 3.4 years) and were treated as an independent data points. Aided sentence testing in noise was completed in only 38% of all cases and 22.2% of the cases that did not meet criteria in quiet. Due to the inconsistency of aided testing in noise, only scores obtained in quiet were included in the analysis.

TABLE 1

Demographic and audiologic data for subjects included in logistic regressions

	Mean (SD)	Minimum	Maximum
Gender (% female)	49.2	—	—
Age (years)	62.5 (18.3)	21	95.2
Better‐ear PTA (dB SPL)	79.0 (17.1)	41.7	121.67
Worse‐ear PTA (dB SPL)	92.2 (18.1)	51.7	121.67
Better‐ear SDS (%)	39.7 (29.3)	0	100
Worse‐ear SDS (%)	18.4 (23.1)	0	96
Best AzBio score (%)	39.4 (30.8)	0	100

Abbreviations: PTA, pure‐tone average (500, 1000, 2000 Hz); SDS, speech discrimination score.

Audiometric measures

Audiometric data included the following: lowest (better‐ear) unaided 3‐frequency (500, 1000, 2000 Hz) PTA (dB HL), highest (better‐ear) unaided percent correct speech discrimination score (SDS) on NU‐6 word list, and highest/best (right‐ear, left‐ear, or binaural) aided AzBio sentence score in quiet administered at 60 dB SPL. Pure‐tone thresholds were obtained at 250, 500, 1000, 2000, 3000, 4000, 6000, and 8000 Hz. Patients with either poorly calibrated hearing aids or no hearing aids were given programmed loaners for use during aided testing. The highest AzBio scores were determined following assessment of the right, left, and bilateral aided conditions. AzBio scores were used to determine candidacy in the models based on criteria for traditional insurers (≤60%) for all subjects and based on Medicare criteria (≤40%) for the subgroup analysis of those ≥65 years old.

Statistical analysis

Two logistic regressions were performed to create models for the predicted probabilities of meeting CI candidacy criteria based on private insurance criteria (best AzBio score ≤ 60%) and for Medicare criteria subgroup (best AzBio score ≤ 40%). For both analyses, better‐ear unaided 3‐frequency PTA and better‐ear unaided SDS scores were entered as predictor variables. Hosmer‐Lemeshow tests were used to determine the goodness‐of‐fit. Plots of true positive rates vs false negative rates for all possible probability cutoff values (ie, receiver operating characteristics [ROC] graphs) were used to assess the performance of the classifiers for distinguishing between potential candidates and non‐candidates. Because this preliminary study ultimately aims to provide clinicians with a tool that can be used as a guide for when to refer, confusion matrices were developed to illustrate how various probability cutoff values affect the sensitivity and positive predictive values (PPVs) of the models. A confusion matrix, also known as an error matrix, is a type of contingency table that depicts an algorithm's performance. In the current study, the performance of the authors' screening tool is visualized based on cutoff value.

RESULTS

Of the 252 patients, 168 patients (66.7%) were found to meet traditional criteria, scoring ≤ 60% on AzBio in quiet. The model predicting candidacy with traditional criteria was statistically significant (Wald's χ 2[2] = 57.07, P < .001) and is shown in Table 2A. Better‐ear PTA (P = .003) and better‐ear SDS (P < .001) were both significant and included in the predictive model. Figure 1A shows the ROC curve displaying combinations of correct and incorrect predictions based on cutoff values ranging from 0.0 to 1.0. The area under this ROC curve is 0.763, indicating the general efficiency of the model.

TABLE 2

Logistic regression results for predicting cochlear implant candidacy in quiet using traditional criteria (AzBio sentence recognition scores ≤ 60%, A) or using Medicare criteria (AzBio scores ≤ 40% in quiet, B)

(A)
							Odds ratio 95% CI
Predictor	β	SEβ	Wald's χ 2	df	P	Odds ratio	Lower	Upper
Constant	−0.523	0.891	0.345	1	.557	0.593
Better PTA	0.031	0.010	8.624	1	.003	1.031	1.010	1.053
Better SDS	−2.485	0.581	18.284	1	.000	0.083	0.027	0.260

FIGURE 1

Receiver operating characteristic (ROC) curves illustrating the trade‐offs between true and false positive rates. Area under the curve = 0.763 for traditional criteria, A and 0.820 for Medicare criteria, B

Table 3A gives a detailed look at the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and total accuracy rates for a range of probability cutoff values. Using a cutoff value of 0.5 (ie, at least a 50% probability for candidacy), the sensitivity was 91% and was 75%. In other words, applying the PTA and SDS values with a 0.5 cutoff could miss 9% of potential candidates (false negatives) and result in negative CIEs for 25% of referred patients. Reduction of the referral criteria to ≥40% probability of candidacy (ie, use of a cutoff value of 0.4) increases the sensitivity to 96% and decreases the PPV to 70%. Probabilities determined by individual PTA and SDS scores are demonstrated in Table 4A

TABLE 3

Confusion matrices for predicting cochlear implant candidacy in quiet using traditional criteria (AzBio sentence recognition scores in quiet ≤ 60% (A), or using Medicare criteria (AzBio scores ≤ 40% in quiet (B), detailing performance of the classification model based on a range of cutoff values. For example, in 3A, changing cutoff value from 0.4 to 0.6 (see bold values), sensitivity decreased from 94% to 78%, respectively, and specificity increased from 29% to 63%

(A)
Cutoff value	Sensitivity	PPV	Specificity	NPV	Accuracy
0.1	100.0	66.7	0.0	—	66.7
0.2	100.0	67.7	4.8	100.0	68.3
0.3	99.4	69.9	14.3	92.3	71.0
0.4	94.0	72.5	28.6	70.6	72.2
0.5	90.5	74.9	39.3	67.3	73.4
0.6	78.0	80.9	63.1	58.9	73.0
0.7	60.7	86.4	81.0	50.7	67.5
0.8	39.9	89.3	90.5	42.9	56.7
0.9	18.5	86.1	94.0	36.6	43.7

TABLE 4

Probability table for meeting cochlear implant candidacy in quiet using traditional criteria (AzBio sentence recognition scores ≤ 60%, (A) or using Medicare criteria (AzBio scores ≤ 40% (B). Bold values are for a selected cutoff value of 0.5

(A)
Better PTA (dB HL)
		40	50	60	70	80	90	100	110	120
Better SDS (%)	100	0.145	0.187	0.239	0.299	0.367	0.441	0.518	0.594	0.665
	90	0.178	0.228	0.287	0.353	0.426	0.503	0.579	0.652	0.718
	80	0.218	0.275	0.340	0.412	0.488	0.565	0.638	0.706	0.766
	70	0.263	0.327	0.398	0.473	0.550	0.624	0.693	0.755	0.807
	60	0.314	0.384	0.458	0.535	0.610	0.681	0.744	0.798	0.843
	50	0.370	0.444	0.520	0.596	0.668	0.732	0.788	0.835	0.873
	40	0.429	0.506	0.582	0.654	0.720	0.778	0.827	0.866	0.898
	30	0.491	0.567	0.641	0.708	0.768	0.818	0.859	0.893	0.919
	20	0.553	0.627	0.696	0.757	0.809	0.852	0.887	0.914	0.936
	10	0.613	0.683	0.746	0.800	0.844	0.881	0.909	0.932	0.949
	0	0.670	0.734	0.790	0.836	0.874	0.904	0.928	0.946	0.960

In the subgroup of 86 potential Medicare beneficiaries, 50 patients (58.1%) were observed to meet Medicare criteria, scoring ≤ 40% on AzBio in quiet. The model for this subgroup was also statistically significant (Wald's χ 2[2] = 32.1, P < .001) and values are shown in Table 2B. PTA (P = .001) and better ear SDS (P < .009) were both statistically significant and included in the predictive model. Figure 1B shows the ROC curve displaying all combinations of correct and incorrect predictions based on cutoff values ranging from 0.0 to 1.0. The area under the ROC curve was .820 (95% CI: .734‐.907), indicating a similar efficiency as the model for traditional candidacy. Table 3B depicts the sensitivity, specificity, PPV, NPV, and total accuracy for a range of cutoff values. Probabilities determined by individual PTA and SDS scores are demonstrated in Table 4B.

DISCUSSION

The current study illustrates the opportunity to use predict candidacy based on probability cutoff classifications to better inform individual patients on their likelihood of CI candidacy. The logistic regression models were developed as proof‐of‐concept for the development of an easy‐to‐ use, tangible “referral calculator” for clinicians. This method is distinct from the previously published recommendations for CI referral, which recommend referral if a patient meets specific audiologic qualifications. Clinicians can consult the values provided in Tables 3 and 4 to determine if a patient's probability for meeting candidacy is acceptable enough to result in a referral and consult the values in Table 3 to derive additional information regarding such a referral. If a particular cutoff value is set as a screening criterion, patients with probabilities exceeding this classification threshold would be recommended for referral. For example, if a screening tool for CI referral incorporated a 0.7 cutoff value (ie, 70% probability or greater of candidacy), the clinician can consult Table 3A to obtain additional information regarding such a referral (specificity = 81, sensitivity = 60.1, PPV = 86.4, and NPV = 50.7). Alternatively, patient‐specific audiologic measurements may also be used to determine the probability: if a patient presents with a better ear SDS of 30% and a better ear PTA of 70 dB, the clinician can consult Table 4A and determine that the patient has a 70% probability of meeting candidacy. Table 3A would thus result in the same specificity, sensitivity, PPV, and NPV (Table 3A). To best meet the needs of patients with moderate‐to‐profound SNHL and address the global underutilization of CIs, an ideal cutoff value should prioritize sensitivity (ie, lowering the probability threshold). As a result, however, this will simultaneously increase the false positive rate of referrals (ie, reduce PPV). If a CI center has limited resources to accommodate increased CIE volume, the center can increase the cutoff value to prioritize a reduced rate of false positive referrals, even if it increases the “misses” (ie, the false negative rate).

Previous work suggests that the lack of a standardized approach to determining when to refer patients for formal CIE contributes to under‐referral and low rates of CI utilization. , Published recommendations may be interpreted differently based on institutional protocols, which vary in their rigidity. For example, if Medicare accepts patients with “bilateral moderate‐to‐ profound SNHL,” the referring provider could interpret this to mean at least moderate vs moderate at lower frequencies and profound at higher frequencies. For the purposes of this study, Medicare criteria for pure tone audiometry were interpreted as requiring all pure‐tone thresholds in the moderate range or greater (≥40 dB HL) with at least one threshold in the profound range (≥90 dB HL) bilaterally. The authors acknowledge that this interpretation of Medicare requirements may be stricter than some institutions. Nonetheless, the model remains significant even when restricting patient inclusion to these pure‐tone threshold requirements. These indistinct guidelines are not limited to Medicare, and varying degrees of hearing loss may be required depending on a patient's insurance provider. A strict application of such criteria may indeed prevent some clinicians from referring individuals who have poor speech perception in conjunction with less‐severe auditory thresholds for a CIE. In contrast, a more liberal interpretation may unnecessarily consume health care dollars and resources by testing too many individuals due to over referral of non‐candidates. However, the authors feel that there is still value to patients and CI programs if a referred patient does not meet CI candidacy. This individual, who otherwise may not have been sent to a tertiary hearing center, will establish care with the CI team, learn more about CI technology, and may become a CI candidate as their hearing loss progresses over time.

Recently, Zwolan and colleagues described a clinically applicable “60/60” referral guideline that yielded a sensitivity of 96%, specificity of 66%, and PPV of 76%. However, the authors of that study acknowledged that patients who meet CI candidacy criteria but have a PTA <60 dB HL or WRS >60% would be missed if strictly using their guideline. Several patients who met the 60/60 guideline also did not meet CI candidacy criteria, reflecting the compromised specificity. This poor specificity is further seen when implementing this tool in our dataset. For private insurance, application of the 60/60 guideline yielded a 45.8% probability of meeting candidacy criteria (Table 4). Application of a 0.4 cutoff to capture the cases meeting the 60/60 criteria then has a sensitivity of 94%, specificity of 28.6%, and PPV of 75% (Table 3). One would expect some level of discrepancy, as our dataset is notably restricted to candidacy results based on sentence testing in quiet, while the study by Zwolan et al routinely incorporated a +10 signal‐to‐noise (SNR) when patients scored > 40% on AZBio sentences. However, the sensitivity and PPV when examining potential candidates with private insurance were comparable. The authors of the current study believe that the logistic regression approach complements the 60/60 referral guideline well, with the understanding that our tool provides the probability of candidacy in quiet alone. The screening tool provides additional detail regarding the likelihood of candidacy and allows for institutions to adjust cutoff values to achieve a false positive rate with which they are comfortable and have the resources to accommodate.

Study limitations and future directives

There are several limitations of this study as well as areas for future investigation. First, our cohort only consisted of people who had specifically been referred for CIE at a single institution, which introduces an element of selection bias. A closer examination of how referring providers interpret and apply recommendations could be beneficial and would promote awareness and education among health care professionals of when to refer patients for CIE, which could facilitate CI uptake. , , , , , A potential source of variability in studies that use audiologic data as part of a screening tool is the quality of the independent variables. Speech discrimination scores may be obtained, for example, from 25‐ or 50‐word lists. The NU‐6 word list, used in this study, consists of 50 words. To the authors' knowledge and based on retrospective chart review, this word list was used for all patients included in the study. Another limitation is that the predictive models were restricted to patients undergoing CIE with sentence testing in the quiet condition. The incorporation of testing in noise is institution‐dependent and typically performed using a +10 dB or +5 dB SNR. , In our cohort, testing in noise was inconsistently performed and not available for all subjects over the period studied. At the authors' institution, background noise is typically incorporated in a stepwise fashion (ie, testing in noise during CIE was only done when CI candidacy was not met in quiet, and +5 dB SNR was only used if CI candidacy was not met at +10 dB SNR). During the study period, only 22.2% of patients who tested negative for candidacy in quiet were subsequently tested in noise (+10 dB SNR). Of this subset tested in noise, 79% were found to be positive for candidacy according to traditional criteria and 73% were candidates according to Medicare criteria when tested in noise. A model that includes sentence testing in noise for only a subset of the sample would thus be inappropriate and a larger dataset will be required to develop logistic regression models that predict CI candidacy in noise. The authors do not recommend against testing in noise after a patient fails in quiet and strongly feel that many patients who fail to qualify in quiet will ultimately benefit from cochlear implantation. Background noise in these cases helps establish candidacy for patients struggling with conventional amplification. ,

Despite these limitations, the current study serves as a proof of concept that demonstrates the feasibility of using logistic regression models to predict CI candidacy based on routine unaided audiometric measures. One would expect that a patient's probability of candidacy would increase with background noise, and this must be kept in mind when the current screening tool determines non‐candidacy based on a given PTA and SDS score. A larger study that includes the routine incorporation of sentence testing in noise in CIEs, as well as use of a strict, well‐defined requirement for pure‐tone threshold requirements, is currently in progress. Through collaboration with multiple CI centers, the performance of such models will be further enhanced, and clinicians will gain a greater understanding of their practical clinical utility.

CONCLUSIONS

A novel screening tool was successfully developed with the intention of aiding audiologists and physicians in determining whom to refer for CIE based on the probability of being a candidate. This preliminary model serves as a proof of concept and is distinct from previously reported referral recommendations in that the cutoff thresholds for determining referrals can be tailored to either the referring provider's or the CI center's practice philosophy regarding an acceptable false‐positive rate. Ideally, the use of this screening tool will help cast a wider net without compromising accuracy, so that patients who would otherwise not be referred for CIE are appropriately evaluated. Lastly, the screening tool provides a patient‐specific probability of candidacy, which facilitates a personalized approach for clinicians and patients to make a more informed decision about their care.

CONFLICT OF INTEREST

C. R. and A. C. M. have received grant support from Cochlear Americas for an unrelated investigator‐initiated research project. C. R., K. V., and A. C. M. serve as paid consultants for Cochlear Americas and Advanced Bionics. A. C. M. serves as CMO and on the Board of Otologic Technologies.