Clinical investigation of speech signal features among patients with schizophrenia.

BACKGROUND: A new area of interest in the search for biomarkers for schizophrenia is the study of the acoustic parameters of speech called 'speech signal features'. Several of these features have been shown to be related to emotional responsiveness, a characteristic that is notably restricted in patients with schizophrenia, particularly those with prominent negative symptoms. AIM: Assess the relationship of selected acoustic parameters of speech to the severity of clinical symptoms in patients with chronic schizophrenia and compare these characteristics between patients and matched healthy controls.
METHODS: Ten speech signal features-six prosody features, formant bandwidth and amplitude, and two spectral features-were assessed using 15-minute speech samples obtained by smartphone from 26 inpatients with chronic schizophrenia (at enrollment and 1 week later) and from 30 healthy controls (at enrollment only). Clinical symptoms of the patients were also assessed at baseline and 1 week later using the Positive and Negative Syndrome Scale, the Scale for the Assessment of Negative Symptoms, and the Clinical Global Impression-Schizophrenia scale.
RESULTS: In the patient group the symptoms were stable over the 1-week interval and the 1-week test-retest reliability of the 10 speech features was good (intraclass correlation coefficients [ICC] ranging from 0.55 to 0.88). Comparison of the speech features between patients and controls found no significant differences in the six prosody features or in the formant bandwidth and amplitude features, but the two spectral features were different: the Mel-frequency cepstral coefficient (MFCC) scores were significantly lower in the patient group than in the control group, and the linear prediction coding (LPC) scores were significantly higher in the patient group than in the control group. Within the patient group, 10 of the 170 associations between the 10 speech features considered and the 17 clinical parameters considered were statistically significant at the p<0.05 level.
CONCLUSIONS: This study provides some support for the potential value of speech signal features as indicators (i.e., biomarkers) of the severity of negative symptoms in schizophrenia, but more detailed studies using larger samples of more diverse patients that are followed over time will be needed before the potential utility of such acoustic parameters of speech can be fully assessed.

Introduction

Schizophrenia is a complex mental disorder caused by multiple factors including heredity, development, and environment.[ The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5)[ lists the following five prominent psychopathological characteristics of the disorder: delusions, hallucinations, disorganized speech, grossly disorganized or catatonic behavior, and negative symptoms. Negative symptoms include emotional blunting, poverty of speech, avolition, an inability to experience pleasure, and the lack of desire to form relationships. Present methods for determining the diagnosis and for assessing the effectiveness of treatment primarily rely on the subjective judgment of the clinician who uses information provided by family members, the mental status examination, and various clinical symptom scales. In the absence of objective measures and the frequent uncooperativeness of patients - particularly those with prominent negative symptoms - assessing the severity and course of the illness is often challenging for clinicians. To address this fundamental problem, psychiatric researchers are actively searching for objective biomarkers that can be used both in the diagnosis of the condition and in the monitoring of the clinical progress of the disorder.

Fluctuations in speech that parallel patients' physio-psychological state might be suitable candidates as biomarkers for schizophrenia. Studies of signal processing and artificial intelligence find that the features of speech signals can contain substantial emotional information.[ Changes of emotions and the range and variability of emotions can be quantified by changes in speech parameters, particularly by changes in prosody - that is, the vocal pitch (fundamental frequency), loudness (acoustic intensity), and rhythm (phoneme and syllable duration) of speech. For example, when a person is in an angry state, changes in physiological characteristics (e.g., increased heart rate, elevated skin voltage, and elevated blood pressure) are often associated with changes in the rate, volume, and tone of speech.

There is considerable interest in developing methods for extracting the acoustic parameters which reflect emotions from speech samples and in assessing the relationship of these parameters to emotionally restrictive states, such as the negative symptoms of schizophrenia. Identification of the emotional content of speech signals is primarily accomplished by two processes: first, the features of the speech signals are extracted from speech samples and then judgments are made about the emotional content of the identified features based on pre-existing models. The quality of the extraction process largely determines the functional quality of the speech identification system.[ Studies about speech identification generally start by investigating the prosodic features and acoustic characteristics of speech content, focusing on the features which are directly relevant to the emotional characteristics of speech.[

Patients with schizophrenia who have prominent negative symptoms such as emotional blunting and poverty of speech may be particularly prone to having restricted emotional content in their speech content. This can directly limit their social functioning and make it difficult for clinicians to detect changes in their clinical status over the course of their illness. Identification of the specific speech abnormalities in such patients could both help in monitoring the course of the illness and potentially be used to develop targeted interventions for patients with prominent negative symptoms. Several researchers[ have reported relationships between specific phonetic parameters and the negative symptoms and impaired emotional perception of schizophrenia, but the results to date are far from robust.

The current study assessed the characteristics of the speech signals of patients with schizophrenia with prominent negative symptoms, considered the association between these features and the severity of different types of negative symptoms, and compared the speech signal features in these patients with those in healthy control subjects.

Methods

Participants

The patient group consisted of patients with schizophrenia who were inpatients at the Shanghai Mental Health Center from September 2013 to December 2015. The inclusion criteria were as follows: (a) aged from 18 to 65 years; (b) met the diagnostic criteria of schizophrenia specified in DSM-5[ as assessed by a psychiatrist using the Mini-International Neuropsychiatric Interview (M.I.N.I. 6.0).[ (c) had prominent negative symptoms of schizophrenia; (d) a minimum of duration of illness of two years; (e) no co-morbid psychiatric or substance abuse disorder; (f) no evidence of severe impulsivity; (g) not using antipsychotic medication that could impair speech; and (h) both the patient and the patient's family member provided written informed consent to participate in the study, including the use of smartphones to record speech.

We recruited volunteers from the community by advertisement as healthy controls. Volunteers were similar for patients in age and duration of education, underwent a through psychiatric exam (using the M.I.N.I. 6.0) and physical exam. Inclusion criteria for controls were as follows (a) 18 to 65 years of age; (b) Han Chinese ethnicity; (c) no current or past physiopsychological, substance abuse, or serious neurological disorder; (d) no serious physical illness; (e) no history of severe impulsivity; (f) no history of suicide attempt; (g) not using antipsychotic medication that could affect speech; (h) no family history of psychiatric disorder or serious neurological disorder; and (i) provided written informed consent to participate in the study.

Measures

We explored a smartphone APP which could record the participant's outgoing speech (i.e., no incoming speech is captured or recorded). Each participant was provided with a preloaded Samsung GALAXY Mega 6.3 (a sampling frequency of 44 kHz and a resolution of 32 bit). Participants (both patients and controls) sat comfortably in a noise-controlled room (background sound below 30 dB) and were asked to use the specially designed smartphone to call a psychiatrist from the study and speak naturally for 15 minutes about any topic of interest.

All call samples were saved in the Advanced Audio Coding (aac) formant. After pre-processing of the data, speech features of interest were extracted and analyzed at the School of Electronic Information and Electrical Engineering of Shanghai Jiao Tong University. We extracted speech features with high emotional identification rates that were identified in previous reports of speech models of bipolar disorder[ and from our own speech model for schizophrenia generated from data in the current study. These parameters include prosodic features (formant 1 to 6 [F1 to F6, unit: Hz], formant bandwidth [unit: Hz], formant amplitude [unit: dB]), and two spectral features (the linear prediction coding [LPC], and the Mel-frequency cepstral coefficient [MFCC]). To assess the stability of the speech data extracted by the software, participants in the patient group repeated the phone call 1 week after the baseline assessment.

In addition, trained attending psychiatrists administered the Positive and Negative Syndrome Scale (PANSS)[, the Scale for the Assessment of Negative Symptoms (SANS), [ and the Clinical Global Impression- Schizophrenia Scale (CGI-S)[ to the participating patients at baseline and 1 week after baseline.

Statistical analysis

All data were processed and analyzed using SPSS 17.0 software. The in-group comparisons from baseline to 1-week post-baseline were analyzed by paired t-tests. The test-retest reliability of the acoustic parameters was assessed by comparing the baseline and 1-week results using intraclass correlation coefficients (ICCs). For between-group comparisons, continuous data with normal distributions were analyzed using independent t-tests; non-normal continuous variables were analyzed using Mann-Whitney U tests; and nominal data were analyzed using Chi-square tests. In the patient group, we use Pearson correlation analyses to assess the strength of the relationship between the acoustic parameters and the severity of clinical symptoms. All statistical analyses used two-tailed tests and statistical significance was set at p<0.05.

Results

As shown in Figure 1, 26 patients completed the two assessments. These patients included 16 males (61.5%); their mean (sd) age was 43.3 (10.9) years; their mean years of education was 9.5 (3.0) years; the mean course of their illness was 21.7 (8.5) years; the mean number of psychiatric admissions was 3.4 (2.4) admissions; and the mean total length of hospitalization was 7.0 (5.4) years. A total of 30 control subjects completed the phonetic assessment; they included 16 males (53.3%), had a mean (sd) age of 37.0 (14.3) years and had a mean duration of education of 11.6 (2.5) years. Comparison between the 26 patients who completed the assessment and the 30 controls who completed the assessment found no statistically significant differences by gender (χ2=0.38, p=0.536), by age (t=1.70, p=0.098), or by duration of education (t=1.95, p=0.058).

Figure 1

Flowchart of the study

As shown in Table 1, in the patient group there were no statistically significant differences between the baseline and 1-week assessment of CGI-S, PANSS total and subscale scores, and SANS total and subscale scores. Thus the patients' clinical status was stable over the 1-week interval.

Table 1

Comparisons of clinical symptoms in 26 patients with schizophrenia at baseline and after 1 week

	baseline mean (sd)	after 1 week mean (sd)	paired t-test	p-value
CGI-S	4.69 (0.74)	4.73 (0.78)	1.00	0.327
PANSS	---	---	---	---
total score	69.23 (9.62)	68.96 (9.64)	1.16	0.258
positive symptoms score	8.15 (1.54)	8.23 (1.53)	1.44	0.161
negative symptoms score	28.23 (4.03)	27.81 (4.24)	2.03	0.054
general psychopathology score	32.88 (5.83)	32.96 (5.86)	1.00	0.327
SANS	---	---	---	---
total score	76.00 (7.43)	74.92 (9.83)	0.87	0.391
affective blunting score	26.27 (3.77)	24.38 (4.04)	1.94	0.064
alogia score	14.15 (1.83)	14.04 (2.36)	0.37	0.713
avolition score	15.92 (2.11)	16.42 (2.63)	1.64	0.114
anhedonia score	19.27 (3.62)	19.73 (3.53)	1.95	0.063
attention score	0.42 (1.03)	0.38 (0.85)	1	0.327

PANSS, Positive and Negative Syndrome Scale[

SANS, Scale for the Assessment of Negative Symptoms[

CGI-S, Clinical Global Impression-Schizophrenia scale[

Table 2 shows the baseline and 1-week results for the acoustic parameters in the patient group and the baseline acoustic parameters in the control group. The test-retest reliability of these measures (only assessed in the patient group) was good, with ICC values ranging from 0.55 to 0.88. The prosody features and formant amplitude and bandwidth were not significantly different between patients and controls at baseline, but the two spectral features were different between the groups: MFCC was significantly lower in the patient group than in the control group and the LPC was significantly higher in the patient group than in the control group.

Table 2

Comparisons of speech features at baseline and after 1 week in the patient group and between patient and control groups at baseline

phonetic parameter	baseline patient group result (n=26) mean (sd)	patient group result after 1 week (n=26) mean (sd)	test-retest reliability of patient results ICC (p-value)	baseline control group result (n=30) mean (sd)	comparison of baseline patient v. control group results t (p-value)
F1 (Hz/dB)	0.036 (0.007)	0.237 (0.028)	0.76 (0.002)	0.040 (0.005)	1.78 (0.081)
F2 (Hz/dB)	0.040 (0.052)	0.084 (0.009)	0.68 (0.009)	0.083 (0.009)	0.95 (0.353)
F3 (Hz/dB)	0.174 (0.015)	0.179 (0.014)	0.84 (<0.001)	0.182 (0.012)	1.43 (0.153)
F4 (Hz/dB)	0.265 (0.023)	0.264 (0.019)	0.67 (0.011)	0.257 (0.019)	0.35 (0.725)
F5 (Hz/dB)	0.359 (0.020)	0.359 (0.014)	0.88 (<0.001)	0.357 (0.013)	0.21 (0.836)
F6 (Hz/dB)	0.426 (0.011)	0.426 (0.012)	0.55 (0.045)	0.426 (0.015)	0.06 (0.951)
formant bandwidth (Hz)	18.83 (11.05)	21.26 (12.41)	0.63 (0.019)	18.10 (9.81)	1.16a(0.247)
formant amplitude (dB)	0.043 (0.005)	0.042 (0.004)	0.61 (0.024)	0.041 (0.009)	1.01 (0.317)
MFCC	-0.085 (0.500)	-0.027 (0.193)	0.87 (<0.001)	0.236 (0.043)	4.97 (<0.001)
LPC	0.249 (0.067)	0.035 (0.006)	0.72 (0.004)	-0.203 (0.367)	5.69 (<0.001)

ICC, Intraclass Correlation Coefficient MFCC, Mel-frequency cepstral coefficient LPC, linear prediction coding

a The homogeneity of variances tests showed that the data were heterogeneous, so comparison of the results of the two groups used the Mann Whitney U test; this is the Z-value of the Mann-Whitney U

Table 3 shows the correlation of 17 clinical and demographic measures with the 10 acoustic parameters in the 26 patients. Among the 170 associations considered, ten coefficients were ≥0.40 and, thus, statistically significant at the p<0.05 level: Formant 1 was negatively correlated with the SANS alogia subscale score, Format 2 was negatively correlated with the PANSS negative symptoms subscale score and the SANS alogia subscale score; Formant 6 was significantly more prominent in male than female respondents; bandwidth was negatively correlated with the SANS affective blunting subscale score and stronger in female respondents than in male respondents; and MFCC was positively correlated with the PANSS general psychopathology subscale score and with the patients' total number of hospitalizations, and it was more prominent in male respondents than female respondents.

Table 3

Correlation of demographic characteristics and the severity of clinical symptoms (at baseline) with the speech features in 26 patients with schizophrenia (Pearson's r)

	F1	F2	F3	F4	F5	F6	bandwidth	amplitude	MFCC	LPC
CGI-S	-0.28	-0.2	0.04	-0.1	0.03	-0.13	-0.13	0.19	0.04	-0.03
PANSS
total score	-0.3	-0.38	-0.16	-0.26	-0.003	0.15	-0.14	0.08	0.31	0.09
positive symptoms score	0.11	0.04	0.09	0.18	0.23	-0.12	-0.23	-0.25	-0.03	-0.36
negative symptoms score	-0.23	-0.40a	-0.21	-0.3	0.05	0.10	-0.01	0.17	0.13	0.26
general psychopathology score	-0.36	-0.36	-0.15	-0.28	-0.11	0.21	-0.16	0.08	0.43a	0.06
SANS
total score	-0.09	0.07	-0.04	-0.18	0.04	0.02	-0.20	0.002	0.06	0.02
affective blunting score	-0.04	0.28	-0.3	-0.50a	0.07	0.31	-0.41a	-0.001	0.15	-0.12
alogia score	-0.42a	-0.42a	0.17	-0.01	0.10	0.23	0.06	0.26	0.20	-0.11
avolition score	-0.04	0.12	0.05	0.08	-0.02	-0.17	-0.04	0.08	0.02	0.09
anhedonia score	0.09	0.1	0.09	0.11	0.02	-0.24	-0.06	-0.18	-0.03	0.12
attention score	-0.04	-0.34	0.22	0.08	-0.22	-0.23	-0.23	0.06	-0.24	0.18
Age	-0.05	0.22	0.28	0.05	0.17	-0.27	-0.09	-0.18	-0.04	-0.05
Gender (1=female, 2=male)	-0.18	0.18	-0.08	0.03	-0.19	0.45a	-0.50b	0.20	0.69b	-0.08
Years of education	-0.18	-0.22	-0.10	-0.003	0.179	-0.25	0.17	0.11	0.07	0.04
Duration of illness	0.07	0.30	0.16	-0.01	-0.04	-0.25	-0.25	-0.21	-0.01	0.01
Number of hospitalizations	-0.23	-0.13	0.10	-0.16	-0.01	-0.06	-0.26	-0.27	0.40a	-0.06
Total time hospitalized	-0.04	0.10	0.10	0.18	-0.11	-0.08	-0.13	0.08	0.22	0.09

PANSS, Positive and Negative Syndrome Scale[

SANS, Scale for the Assessment of Negative Symptoms[

CGI-S, Clinical Global Impression-Schizophrenia scale[

MFCC, Mel-frequency cepstral coefficient

LPC, linear prediction coding

a 0.01

b 0.001

Discussion

Main findings

We found that when the severity of psychiatric symptoms remains stable, the speech features selected to assess the emotional content of the voice samples of patients with schizophrenia with prominent negative symptoms were also stable over a 1-week period. Correlation analyses of these measures with clinical and demographic characteristics of the patients identified several potentially important relationships, a finding that has been reported in previous studies.[ Comparison of these speech features between patients and matched healthy controls found no statistically significant differences in the prosody features or formant bandwidth and amplitude, but there were significant differences in the two spectral features considered: the MFCC was significantly lower in patients than controls, while the LPC was significantly higher in patients than controls.

Other studies have reported that these two spectral features play an important role in everyday communications.[ Spectral features have also been found to be useful for discriminating emotions in artificial intelligence studies. The LPC is a relatively efficient and accurate measure of the waveform and spectrum of speech that is used in speech coding, speech synthesis, speech identification, and other applications.[ The MFCC, which modifies external signals in a manner similar to the human ear, is a reliable parameter for discriminating different emotional states.[ Similar to our results, a study by Sun and colleagues[ found that (when using a sorter based on a Gaussian mixture model [GMM] algorithm) the MFCC was better at discriminating different emotional states than the prosody features. Further work is needed to determine whether or not these spectral features can be used as biomarkers for the identification and monitoring of schizophrenia or of the prominent negative symptom subtype of schizophrenia.

The correlation analysis identified some intriguing associations between 6 of the 10 speech signal features considered (F1, F2, F4, F6, bandwidth, and MFCC) and 6 of the 17 clinical and demographic parameters considered (gender, the PANSS negative symptoms and general psychopathology subscale scores, the SANS affective blunting and alogia subscale scores, and the number of hospitalizations). Other studies have also identified significant correlations between different speech features and the negative symptoms of schizophrenia.[ This raises the possibility that a subset of acoustic parameters of a standardized speech sample - potentially transmitted over a smart phone to clinicians - could be used to either monitor the severity of negative symptoms or predict the subsequent course of the illness. However, given the small sample size and the large number of potential associations considered in the current study, these results need to be replicated before they can be meaningfully interpreted.

Limitations

The present study has several limitations that need to be considered when interpreting the results. The speech features selected may not be the most sensitive measures of changes in chronic schizophrenia; further research using a wider range of measures will be needed to find other, potentially more sensitive, measures. The 1-week interval we used to assess the test-retest reliability of the speech features indicated short-term stability in the phonetic parameters, but we are uncertain how stable such measures are over a longer time period. A total of 170 correlations between the 10 speech features assessed and 17 clinical and demographic characteristics are considered, so the statistically significant relationships identified may be chance findings; repeat studies are needed to confirm their importance. The comparison between patients and controls was cross-sectional so we cannot report on the sensitivity of the speech features to changes in clinical symptoms; longitudinal studies that compare changes in the speech features to changes in the clinical measures will be needed to determine their potential utility as biomarkers of clinical changes. All the patients included in the study had a prolonged course of illness and had been on antipsychotic medication for many years, so it is possible that this may have had an effect on the assessed speech features.[ Finally, the sample was quite small - only 26 patients - so some of the negative findings (e.g., failure to identify differences between patients and controls) may have been due to Type Ⅱ errors.

Importance

This study focused on the negative symptoms of schizophrenia, symptoms that are often not improved with standard antipsychotic medications and that often predict a poor prognosis and progressive deterioration in social functioning.[ The study is a preliminary assessment of the feasibility of using speech features that assess the emotional characteristics of speech as biomarkers for the severity of negative symptoms in schizophrenia and, thus, as potential predictors of the prognosis of the disorder. The selected speech features included both the prosodic variables used in prior studies (i.e., rate, volume, rhythm, etc.) and two spectral features (MFCC and LPC) that have previously been shown to be useful in the emotional characterization of speech samples. These speech features proved to be stable (over a short period), some of them - the spectral features rather than the prosody features reported in some previous studies - were significantly different between patients and controls, and some of them were significantly correlated with clinical measures of negative symptoms. However, this was a cross-sectional study in a small group of chronic patients, so much more detailed studies using larger samples of more diverse patients that are followed over time will be needed before the potential utility of such speech features can be fully assessed.