Noninvasive Vocal Biomarker is Associated With Severe Acute Respiratory Syndrome Coronavirus 2 Infection.

OBJECTIVE: To investigate the association of voice analysis with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. PATIENTS AND METHODS: A vocal biomarker, a unitless scalar with a value between 0 and 1, was developed based on 434 voice samples. The biomarker training was followed by a prospective, multicenter, observational study. All subjects were tested for SARS-CoV-2, had their voice recorded to a smartphone application, and gave their informed consent to participate in the study. The association of SARS-CoV-2 infection with the vocal biomarker was evaluated.
RESULTS: The final study population included 80 subjects with a median age of 29 [range, 23 to 36] years, of whom 68% were men. Forty patients were positive for SARS-CoV-2. Infected patients were 12 times more likely to report at least one symptom (odds ratio, 11.8; P<.001). The vocal biomarker was significantly higher among infected patients (OR, 0.11; 95% CI, 0.06 to 0.17 vs OR, 0.19; 95% CI, 0.12 to 0.3; P=.001). The area under the receiver operating characteristic curve evaluating the association of the vocal biomarker with SARS-CoV-2 status was 72%. With a biomarker threshold of 0.115, the results translated to a sensitivity and specificity of 85% (95% CI, 70% to 94%) and 53% (95% CI, 36% to 69%), respectively. When added to a self-reported symptom classifier, the area under the curve significantly improved from 0.775 to 0.85.
CONCLUSION: Voice analysis is associated with SARS-CoV-2 status and holds the potential to improve the accuracy of self-reported symptom-based screening tools. This pilot study suggests a possible role for vocal biomarkers in screening for SARS-CoV-2-infected subjects.

During the current coronavirus disease 2019 (COVID-19) pandemic and in the absence of any pharmaceutical intervention, the contemporary strategy against disease spread is social distancing. Global strict confinement restrictions have been associated with the reduction of new cases in many countries, turning the attention to a possible “second wave” of the disease. The suppression of new waves of viral infection is dependent on the scope and efficiency of testing strategies. Molecular and serologic testing are the gold standard methods for both identifying infected people and for gathering information on people who have recovered. However, there is a clinical need for remote, noninvasive, and transparent methods to screen large populations in specific scenarios such as airports and public transportation hubs. There is also a clinical need for a self-administered pre-screening tool available to the general public that will significantly improve classification and the effectiveness of the existing polymerase chain reaction (PCR) testing regime.

Voice signal analysis and voice recognition are being used extensively for commercial purposes. Amazon, Google, Samsung, and other companies are using the technology to allow customers to talk, activate, and search their devices for content and it is estimated that one in six Americans owns a voice-activated device. Recent data suggest that voice analysis can be used to develop vocal biomarkers that are associated with disease states. Examples include Parkinson’s’ disease, coronary artery disease, pulmonary hypertension, and chronic obstructive pulmonary disease.3, 4, 5 We have recently shown how vocal biomarkers can identify congestive heart failure patients at risk for hospital re-admission and mortality.

Lungs play a critical role in voice production, and voice may be affected by interstitial fluid and pulmonary edema. Thus, it is biologically plausible that severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection could be detected by voice signal analysis. The purpose of the current pilot study was to investigate the association between voice analysis and SARS-CoV-2 infection in a prospective multicenter clinical registry.

Patients and Methods

Study Population

This is an observational, prospective, multicenter clinical registry of audio recordings and clinical data from patients with SARS-CoV-2 and negative controls. The analysis of the collected data was conducted retrospectively and did not affect the disease management. Data were collected in parallel in three sites in Israel during the first pandemic wave: Sheba Medical Center, Rabin Medical Center, and the Israeli Army Medical Corps. Patients were recruited while being quarantined in a center for mild COVID-19 patients in one of those three sites. Negative subjects in this study were subjects who underwent a clinically indicated SARS-CoV-2 PCR test and were found to be negative. Data collection was approved by the ethical committees of all sites and participants signed an informed consent form before any data was collected. Patients were included in the study if they were (1) capable to comply with the study protocol, handle smartphone application, and have basic computer skills; and (2) signed informed consent. Exclusion criteria included age younger than 18 years, pregnancy, inability to sign informed consent documents, or any speech or voice impairment. Demographic data, medical history, and SARS-CoV-2 test history were documented in pre-specified case report forms. The participants were divided into two groups based on their PCR results. Group A included patients with laboratory-confirmed SARS-CoV-2 infection who were hospitalized or in quarantine outside of hospital (n=40). Group B included participants who underwent a clinically indicated PCR test and were PCR-confirmed negative (n=40).

Voice and Data Collection

Participants were instructed to download the Vocalis Health Research mobile application to record themselves and to document their symptoms at the time of the recording. Participants were instructed to record while holding the phone in their hand or putting the phone on a table in front of them at a distance that allows them to read text on the screen. In addition, participants documented the following symptoms: shortness of breath (not at all/light/mild/severe), cough (not at all/light/mild/severe), runny nose (not at all/light/mild/severe), fever above 37.8°C (y/n/did not check), and decreased smell sensations (yes/no/don't know). Data collected via the study-dedicated patient’s smartphone using the Vocalis Health Research mobile application was uploaded and stored encoded in a secured cloud hosted at Amazon Web Services. The transmission of data from the mobile app to the secured cloud was conducted automatically according to a secured and encoded standard SSL 3/TLS 1.3.

Voice Characteristics and Vocal Biomarker Definition

Development of the vocal biomarker was based on a training cohort of 434 voice samples for a total of 272 participants, of whom 160 (59%) were SARS-CoV-2–positive. Voice samples included healthy volunteers recorded through the Vocalis Health website free application, as well as existing online audio recordings. The mean age of the training cohort was 37±16.3 years and 59% were males. The feature extraction process was based on previously described transfer learning and adaptation methods, which are appropriate for small training databases. All recordings were re-sampled to 16 kHz and a Mel spectrogram was calculated using the Librosa library in Python, window size was 1024 samplers, hop size was 512 samples, and the number of Mel coefficients was set to 128. For each recording, 10 seconds of continuous speech were converted to a Mel spectrogram. Each Mel spectrogram was passed through a Vggish convolutional neural network architecture which was pre-trained on the Vocalis internal databases. This process can be viewed as feature extraction, converting 10 seconds of continuous speech recording to a 512-dimensional features vector (Figure 1). A 10-fold cross-validation procedure was conducted on the training cohort patients to evaluate two classification models (random forest and support vector machine, scikit-learn implementation in Python) at different regularization levels via a grid search. The best model was selected using the average of the 10-folds area under the receiver operating curve (AUC) metric. The resulting model is described as the vocal biomarker, a positive scalar between 0 and 1, which is a nonlinear combination of the 512 features mentioned above. The best model on the training cohort achieved an AUC of 0.78±0.08, using a support vector machine model with a nonlinear kernel (radial basis function) and a regularization constant C=1. Because some of the participants contributed more than one voice recording, data leakage between folds must be excluded. Thus, we randomized the labels (positive/negative) between the participants and reached an AUC of 0.5±0.03, which is equal to a random classifier. This randomization validated that there was no data leakage between the various folds in the 10-fold cross-validation process. The biomarker that achieved the highest AUC on the 10-fold cross-validation procedure was tested on the study population.

Figure 1

Feature extraction process using transfer learning and adaptation methods. The process produces a 512-features vector for each x seconds of a given recording. The process is agnostic to the chosen recording length, due to a global average pooling layer at the end of the fine-tuned Vggish network.

Statistical Analysis

Study population was divided into positive and negative groups for SARS-CoV-2 based on the PCR test result. Variables were expressed as median with interquartile range and as frequency (%) for categorical variables. The two population groups were compared using a Mann-Witney test for continuous variables, and a χ2 test for categorical variables. The diagnostic accuracy of the vocal biomarker in identifying SARS-CoV-2 status was evaluated and graphically displayed using receiver operating characteristic (ROC) curve and its subsequent AUC. The statistical analyses were performed with Python programming language (Python Software Foundation, https://www.python.org) and IBM SPSS Statistics software for Windows, version 23.

Results

The final study population included 80 subjects of whom 40 (50%) were positive for SARS-CoV-2 infection. Median age of the study population was 29 (range, 23 to 36) years and 54 (68%) were men. The overall rate of comorbidities in the study cohort was relatively low: there were 8 (10%) active or past smokers, 3 (4%) patients with asthma, 3 (4%) patients with hypertension, 2 (3%) patients with diabetes mellitus and 2 (3%) patients with neurological disease. Patients with SARS-CoV-2 were more likely to have a history of tobacco use and had similar body mass index as PCR-negative subjects. Baseline characteristic of the study population with comparison between the two study groups are summarized in Table 1. Self-reported clinical symptoms of both study groups are displayed and compared in Table 2.

Table 1

Baseline characteristics of the study populationa,b

Characteristic	All (N=80)	COVID-19 (-)(n=40)	COVID-19 (+)(n=40)	P
Age, years (all data)	29 (23,36)]	29 (24, 35)	28 (22, 37)	.415
Male	54 (68)	25 (63)	29 (73)	.340
Smokers	8 (10)	2 (5)	6 (15)	<0.001
Obesity (BMI > 30 kg/m2)	5 (6)	1 (3)	4 (10)	.358
Asthma	3 (4)	1 (3)	2 (5)	.500
Neurological disease	2 (3)	1 (3)	1 (3)	.747
Hypertension	3 (4)	1 (3)	2 (5)	.510
Diabetes mellitus	2 (3)	0	2 (5)	.253
GERD	1 (1)	0	1 (3)	.506
CKD	1 (1)	0	1 (3)	.506
Treated respiratory conditionc	6 (8)	1 (3%)	5 (13)	.09
Vocal biomarker (all date)	0.14 [0.1-0.28]	0.11 [0.06-0.17]	0.19 [0.12-0.3]	.001
Days between PCR and voice recording (all date)	10 [4-20.3]	19 [9.8-32]	6 [2,10]	.02

BMI, body mass index; CKD, chronic kidney disease; COVID-19, coronavirus disease 2019; GERD, gastroesophageal reflux disease, PCR, polymerase chain reaction.

Values are n (%) unless otherwise stated.

Defined as treated with pulmonary medications.

Table 2

Self-reported symptoms at the time of the recordinga,b

	All	COVID-19 (-)	COVID-19 (+)	P
Fever	4 (5)	0	4 (10)	<.001
Cough	20 (25)	2 (5)	18 (45)	<.001
Shortness of breath	8 (10)	1 (3)	7 (18)	.025
Runny nose	17 (21)	5 (13)	12 (30)	.05
Loss of smell	14 (18)	0	14 (35)	<.001
At least one symptom	33 (41)	6 (15)	27 (68)	<.001

COVID-19, coronavirus disease 2019.

Values are n (%).

SARS-CoV-2 Status and the Vocal Biomarker

When evaluated as a continuous unitless scalar, the vocal biomarker was significantly higher among patients with SARS-CoV-2 (0.19 [0.12-0.3] vs. 0.11 [0.06-0.17], P=.001) (Figure 2). A multivariate binary logistic regression with adjustment for age and sex showed that each 1 unit increase in the vocal biomarker was associated with a six-fold increased likelihood of SARS-CoV-2–positive status (adjusted odds ratio [OR], 6.43, P=.108). To estimate the biomarker variability, the biomarker was calculated for two separate 7-second segments of the same recording for all participants in the study cohort (n=80). This analysis demonstrated strong statistical correlation (Pearson’s correlation coefficient 0.78, P<.001). The ROC curve evaluating the association of the vocal biomarker with SARS-CoV-2 status is displayed in Figure 3, with an AUC of 0.72. With a biomarker threshold of 0.115, the results translated to a sensitivity and specificity of 85% (95% CI, 70% to 94%) and 53% (95% CI, 36% to 69%), respectively (Table 3). A subanalysis excluding 15 positive SARS-CoV-2 patients with more than 7 days between PCR test and voice recording (40 negative and 25 positive patients, n=65) yielded consistent results with an AUC of 0.715. The consistency and robustness of the vocal biomarker was evaluated for varying recording lengths. The vocal biomarker training process was repeated for recording lengths between 3 and 20 seconds. The vocal biomarker was trained on the training cohort, 10-fold cross-validation was performed, and the results of the training cohort and the validation cohort are presented in grey bars in Figure 4. Analyzing less than 6 seconds of voice recording results in degradation in performance, also in the training set AUC but most specifically in the study population AUC, which dropped below 0.70 (P<.001). The range between 10 and 14 seconds shows the best results with an AUC above 0.71 on the study population. The degradation in performance in short voice recordings might be attributed to the lack of temporal information which is crucial to catch the dynamics of this pathology, which in turn affects the vocal biomarker’s ability to distinguish between SARS-CoV-2–negative and –positive patients. On the other hand, as each recording is transformed to a vector of 512 features (Figure 1), at higher lengths of recordings (more than 14 seconds), this process of dimensionality reduction might result in loss of information, thus affecting the AUC results on the study population.

Figure 2

Scatter plot of vocal biomarker values by the two study groups can be seen on the Y-axis.

Figure 3

Receiver operating characteristic (ROC) curves for the vocal biomarker and symptom-based classifier are shown. Study population (n=80) comparison between the average area under the curve (AUC) for coronavirus disease 2019 detection (positive/negative) of the combined biomarker (red, AUC = 0.85), the vocal biomarker (green, AUC = 0.72), and the symptoms-based classifier (blue, AUC = 0.77). The blue represents the symptoms classification result.

Table 3

Vocal biomarker classification compared to PCR resultsa,b

Vocal biomarker	PCR test result		Total
	Positive	Negative
Positive	34	19	53
Negative	6	21	27
Total	40	40	80

PCR, polymerase chain reaction.

The classification results of the Vocalis coronavirus disease 2019 biomarker in comparison to the PCR test results in the study population (n=80) with a threshold of 0.115. The sensitivity of the biomarker was 85% (95% CI, 70.2% to 94.3%), the specificity was 52.5% (95% CI, 36.2% to 68.5%), and the positive predictive value was 64.2% (95% CI, 55.8% to 71.8%) and negative predictive value was 77.8% (95% CI, 61.3% to 88.6%).

Figure 4

Effect of varying the voice window length on the performance of the coronavirus disease 2019 classifier. Between 3 seconds and up to 20 seconds, the grey bars are the resulted average area under the curve (AUC) on the training set cross-validation process. The blue bars are the resulted AUC on the study population (n=80).

Vocal Biomarker Association With Signs and Symptoms

Patients with SARS-CoV-2 were more likely to report fever, shortness of breath, cough, and runny nose (Table 2). Binary logistic regression showed that, compared to SARS-CoV-2–negative patients, SARS-CoV-2–positive patients were 12 times more likely to report at least one symptom (OR, 11.77; 95% CI, 4 to 35; P<.001), were 15 times more likely to report cough (95% CI, 3 to 73, P<.001), and were 8 times more likely to report shortness of breath (95% CI, 1 to 70, P=.05). When patients reporting at least one symptom were classified as positive, this symptom-based classifier reached an AUC of 0.775, which is consistent with previous reports on the accuracy and AUC of symptom-based screening tools. When the vocal biomarker threshold of 0.115 was used, 9 of 34 (26%) true positive patients who were correctly classified by the biomarker reported no symptoms. In addition, in the full study cohort, 9 of 13 (69%) asymptomatic SARS-CoV-2–positive patients were detected with the vocal biomarker. The independent incremental contribution of the vocal biomarker to SARS-CoV-2 screening was confirmed by combining the symptom-reported classifier with the biomarker using the following simple formula:

The resulting combined biomarker, a continuous scalar between 0 and 1, gives equal weight to both the reported symptoms and the voice analysis. This combined biomarker reached an AUC of 0.85. A comparison between the combined biomarker and the vocal biomarker can be seen in the ROC curve in Figure 3. The combined biomarker (red, AUC = 0.85) outperforms both the vocal biomarker (green, AUC = 0.72) and the symptoms indicator (blue, AUC = 0.775).

Discussion

This pilot study has several important observations. First, this is the first clinical observation showing an association between viral respiratory infection and voice analysis, emphasizing the potential feasibility of using voice biomarkers to identify SARS-CoV-2 infection. Second, it shows that at least 6 seconds of continuous speech are crucial to represent the SARS-CoV-2 infection effect on voice. Lastly and most importantly, our analysis suggests that voice analysis might have independent incremental value when added to a self-reported symptom-based screening classifier and can assist in remotely identifying asymptomatic individuals.

In recent years, with the widespread use of wearable devices and smartphones, there is a growing interest in a remote voice analysis as a complimentary noninvasive telemedicine tool. Machine learning algorithms have helped to identify an association between voice and several disease states including coronary artery disease, pulmonary hypertension, and congestive heart failure patients at risk for re-admission and/or death.4, 5, 6 Voice is just one example of the many digital biomarkers that are emerging in recent years due to advances in artificial intelligence and machine learning algorithms, coupled with high-quality big data electronic registries. A recent example is the use of deep learning to detect coronary artery disease based on facial photos with AUC of 0.73. In the specific case of vocal biomarkers, a correlation between respiratory viral infection and changes in voice analysis is physiologically plausible. Voice is created by three major components that include the lungs, the larynx, and the articulators (eg, the tongue, the palate, and the mouth muscles). The current study extends and supports preliminary evidence linking abnormal lung state of pulmonary congestion with changes in voice. In a preliminary study by Murton et al, patients with decompensated heart failure who were successfully treated in hospital demonstrated a higher proportion of automatically identified creaky voice, increased fundamental frequency, and decreased cepstral peak prominence variation, suggesting that speech biomarkers can be early indicators of heart failure. Murton et al suggested a role for vocal cord and lung edema in the changes in voice analysis before and after treatment. This study was followed by a larger more recent study by Amir et al who consistently showed in 40 patients how voice changes between “wet” and “dry” states of patients with acute heart failure. Moreover, our previous study demonstrated that voice biomarkers are associated with pulmonary hypertension. The results of the current analysis are in line with and extend these preliminary observations. We successfully show in our analysis how voice is associated with viral lung disease. Although viral lung disease, pulmonary hypertension, and pulmonary congestion are distinct clinical entities, all share several lung abnormalities including inflammation and edema of the lung parenchyma, increased pulmonary vascular resistance, and changes in expiratory flow capacity.

This analysis has important clinical implication for the remote SARS-CoV-2 status screening. Molecular-based diagnostics remain the gold standard for the diagnosis of SARS-CoV-2 infection. However, with the lifting of the social distancing restrictions, there is a clinical need for simple noninvasive screening tools for allowing a safe lifting of social distancing restrictions. In Israel, for example, the lifting of the social distancing restrictions and re-opening of schools was associated with a “second wave” of infection. The widely used temperature check has proven to be inefficient providing negligible value for disease control. The US Centers for Disease Control and Prevention recommend a simple screening tool based on simple self-reported signs and symptoms that include fever or feeling feverish (chills, sweating), new cough, difficulty breathing, sore throat, muscle aches or body aches, vomiting or diarrhea, and new loss of taste or smell. With the contemporary use of smartphone and wearable devices by so many individuals, voice can be used to assist in remote screening for SARS-CoV-2 infection. As we show in our analysis, voice analysis is associated with SARS-CoV-2 infection and can significantly improve classification when compared to a self-reported symptom-based classifier. Possible clinical uses for such a tool are many and can include hospitals, shopping malls, airports, and public transportation hubs to name a few. Another important clinical use could be for decision support in classifying candidates for PCR testing, and whether they are already required to self-isolate or are experiencing relevant symptoms. In general, the demand for PCR testing among the public far exceeds the supply and current methods for determining eligibility for PCR testing are inaccurate and expensive (when requiring consultation with a doctor). Therefore, there is a dire need to increase this percentage with a readily available and effective pre-screening tool.

Study Limitations

This pilot study has several limitations. First, this is a relatively small observation study focusing on young and relatively healthy subjects and with relatively low specificity. Therefore, further larger studies are needed to increase the biomarker specificity and for this work to be generalized to other populations. Second, this study examined Hebrew-speaking individuals. While the tools we used for voice analysis are agnostics to language and were trained on other languages, there is a need to confirm our findings in other languages as well. Third, although our findings showed a correlation with SARS-CoV-2 infection, it might represent a nonspecific voice change associated with any respiratory viral infection. Lastly, the study did not include time series voice tests for each subject. Further study including time series testing would provide a personalized baseline for each subject that could improve the accuracy of the algorithm.

Conclusion

This is the first study to document a relationship between a vocal biomarker and SARS-CoV-2 respiratory infection. Vocal signal analysis is a noninvasive biomarker that holds the potential to assist in remote screening of large populations that could increase the effectiveness of current PCR testing strategies, both in conjunction with social distancing restrictions and after they are lifted. Together with other screening tools, it holds the potential to assist in fighting the spread of the current COVID-19 pandemic.