Judgement of valence of musical sounds by hand and by heart, a machine learning paradigm for reading the heart.

The intention of the experiment is to investigate whether different sounds have influence on heart signal features in the situation the observer is judging the different sounds as positive or negative. As the heart is under (para)sympathetic control of the nervous system this experiment could give information about the processing of sound stimuli beyond the conscious processing of the subject. As the nature of the influence on the heart signal is not known these signals are to be analysed with AI/machine learning techniques. Heart rate variability (HRV) is a variable derived from the R-R interval peaks of electrocardiogram which exposes the interplay between the sympathetic and parasympathetic nervous system. In addition to its uses as a diagnostic tool and an active part in the clinic and research domain, the HRV has been used to study the effects of sound and music on the heart response; among others, it was observed that heart rate is higher in response to exciting music compared with tranquilizing music while heart rate variability and its low-frequency and high-frequency power are reduced. Nevertheless, it is still unclear which musical element is related to the observed changes. Thus, this study assesses the effects of harmonic intervals and noise stimuli on the heart response by using machine learning. The results show that noises and harmonic intervals change heart activity in a distinct way; e.g., the ratio between the axis of the ellipse fitted in the Poincaré plot increased between harmonic intervals and noise exposition. Moreover, the frequency content of the stimuli produces different heart responses, both with noise and harmonic intervals. In the case of harmonic intervals, it is also interesting to note how the effect of consonance quality could be found in the heart response.

Introduction

The electrocardiogram is a record over time of the heart electrical activity; this activity is registered as an analogue signal known as the electrocardiographic signal (ECG) [1]. Because of its ambulatory character and simplicity in comparison with other medical procedures, ECG is the most common heart medical exam [2]. It is often used as an indicator of the physiological condition and as a diagnostic tool of the cardiovascular system, in particular of the heart health [3]. Among other models of ECG analysis, heart rate variability (HRV), derived from the R-R interval peaks of ECG, emerges to reveal relationships between autonomous neuron system and physiological, physical or psychological variables [4]. HRV exposes the interplay between the sympathetic and parasympathetic nervous system [5]. HRV has been applied to study cardiovascular diseases such as acute ischemic stroke [6], cardiac autonomic dysfunction [7], cardiopulmonary dysfunction [8], myocardial infarction [9], and cardiac death [10]. HRV has been considered to assess panic disorder [11], mental health resilience [12], and depressive disorder [13]. The effects of electronic cigarette [14], exercise [[15], [16]], and alcohol use [17] have been also evaluated using HRV.

In the same way, HRV has also been employed to assess the effects of sound and music on the heart [18], where research on this topic has gained importance in the recent time since it makes it possible to understand and take advantage of the music benefits [[19], [20]], as the decreasing in the heart rate and in the systolic and diastolic blood pressure [21]. Thus, music has been studied from several perspectives. For instance, in the direction of multimodal music information, several tasks have been developed, among them it is possible to find music segmentation, emotion or mood recognition, synchronization of different representations, and classification of music [22]. In like manner, some researchers have developed systems based on deep learning to create music [23]. In addition to fields as Computational Musicology [24] and Interactive music [25], the use of music in health sciences has also been examined to manage several diseases or conditions such as autism [26], depression [27], cancer [28], and cardiovascular diseases [20].

Besides a lot of applications of the ECG and HRV in the clinical domain and research, they have been used to analyse how music with different emotional character affects the heart [[29], [30], [31]], and some of these studies have been carried out with both musicians [32] as non-musicians [29]. Although the effects of music on heart behaviour have been studied, research has not established the best way of audio stimulation [18] and because until now it is still unclear which musical element is related to the observed changes, it is required to develop systematic high-quality research on the effects of music on the heart [33]. In this aspect, most of the previous research has focused on the effects produced by the tempo of music [[34], [35], [36]]. Literature reports a great variation of the effects of music on HRV, avoiding the possibility to draw substantial conclusions [18]. Previous research has classified the HRV response to old generation romantic music with a performance of 80% of accuracy, using artificial neural network [37]. Another research classified the response of the Autonomic Nervous System with the HRV to Odia and Tamil music [38]. The classification used the Regression Tree, Boosted Tree and Random Forest algorithms and a performance of 85% was achieved. From previous studies, it is observed that is required to develop classification systems of HRV responses to sound stimuli to improve the performance of previous studies.

Consequently, this study assesses in more detail the effects of harmonic intervals on the heart response. Additionally, it evaluates the heart reaction to noise stimuli. To achieve this goal, some machine learning techniques were considered to make associations between stimuli and heart responses and compared with the judgements of the subjects on the valence of the sounds. These techniques were selected considering recent experimentations that have done the data analysis throughout artificial intelligence tools [[19], [37], [39], [40]]. This study hypothesizes that the selected sound stimuli can produce different responses in the heart. It is hypothesized that heart responses can be classified according to the sound stimuli using algorithms of machine learning.

The remainder of this document is organized as follows: Section 2 presents a description of the methods where the experimental procedure and data processing are shown; Section 3 reports the results, where the most salient outcomes can be observed; Section 4 covers discussion of the results; and finally, Section 5 shows the conclusions of this study.

Methods

Experimental procedure

Participants

Participants were voluntarily enlisted in the experiment, 26 healthy subjects without vocational music training, 17 males and 9 females, with an average age of 25.3 years old (SD = 7.1), ranged from 18 to 37 years. The participants underwent audiometry exam to ensure they could hear well. They were asked to sleep well, not to consume either caffeine or alcohol, not to practice sports or to consume stimulant substances 24 h before sample collection. All procedures were carried out considering the Declaration of Helsinki to keep the safety and confidentiality of subjects. All procedures, including experimentation on human subjects, were approved by The Internal Ethical Committee of Universidad del Cauca, and the research was done according to the approved protocol.

Sounds

In this experiment, 30 different stimulus conditions of two different types were used: noise, and harmonic music intervals (HMI). The noise and HMI signals were synthesized in order to have as much control as possible in the stimulus presentation. The noise stimuli taken into account were: blue, brown, grey, pink, purple and white noise. The HMI type consisted of the all possible harmonic intervals in one octave, including two different octaves: a lower since A2 to A3 and a higher between A4 and A5. Each musical note was composed of more than one partial frequency, i.e. complex tones; as partial frequencies increased, their power decreased. The power spectrum and timbre of the harmonic sounds synthesized were similar to the flute. All these intervals had the A2 and A4 as the low notes, while the higher ones were changed. Thus, stimuli had the intervals in two octaves, octave 2 and 4: minor second (2m), major second (2M), minor third (3m), major third (3M), perfect fourth (4), augmented fourth (4aug), perfect fifth (5), minor sixth (6m), major sixth (6M), minor seventh (7m), major seventh (7M), octave (8). To avoid the influence of changes in the volume and the intensity sound, the perceived loudness was normalized in all sounds by applying ReplayGain [41]. The responses to the used stimuli were analysed from the following aspects as type (noise and HMI), frequency (high and low), consonance and dissonance (HMI), and as independent stimuli (Table 1).

Table 1

Description of the used stimuli.

Type	Index∗			Description	FrequencyClass	ConsonanceClass	Instances
Harmonic intervals (HMI)	1	-	13	Minor second (2m)		Dissonant	5616 (234 per each stimulus)
	2	-	14	Major second (2M)		Dissonant
	3	-	15	Minor third (3m)		Consonant
	4	-	16	Major third (3M)		Consonant
	5	-	17	Perfect fourth (4)	Low:	Consonant
	6	-	18	Augmented fourth (4aug)	Octave 2	Dissonant
	7	-	19	Perfect fifth (5)	High:	Consonant
	8	-	20	Minor sixth (6m)	Octave 4	Consonant
	9	-	21	Major sixth (6M)		Consonant
	10	-	22	Minor seventh (7m)		Dissonant
	11	-	23	Major seventh (7M)		Dissonant
	12	-	24	Octave (8)		Consonant
Noise			25	Grey	Low and High	---	1404 (234 per each stimulus)
			26	White	Low and High
			27	Brown	Low
			28	Pink	Low
			29	Blue	High
			30	Violet	High
Total							7020

1–12: lower octave; 13–24: higher octave.

In addition to consonant and dissonant analysis, the sounds were studied in three and nine classes according to the perception scores of subjects. In three classes, a negative class was defined with scores between one and three, while the positive class took scores between seven and nine; in addition, a neutral class was defined with scores between four and six. Finally, in nine classes each set of scores was considered as an independent class, i.e. one class per score, from one to nine.

Data collection

The procedure was made in an isolated room from external stimuli, with an average temperature of 23 °C, sound pressure level of 40 dB, and illumination of 100 Lux. Before the procedure, once the research and its purposes were explained, the participants signed the consent form. The experiment was conducted with one subject at a time, who reminded isolated, and in a comfortable stretcher. The subjects were in a rest supine position for 15 min and after this period they were equipped with Bose Noise Cancelling Headphone 700 and they were also asked to close their eyes to avoid the influence of any visual stimuli. At once, a baseline was measured for two minutes. Following this time, 30 sounds were played in random order; 24 sounds with harmonic intervals in the octaves 2 and 4, and six different noise sounds. These sounds were played during ten seconds each and were separated by a silence section of 15 s. The subjects were instructed to score their perception about the listened sound immediately each sound had finished; then they should have opened their eyes, score their perception on a screen projection in front of them, and closed their eyes again. The procedures were designed to avoid the effort of subjects, reducing the need to speak and move. Subjects registered their perception by using a Bluetooth mouse in a user interface, on a scale between 1 and 9, where 1 represented the worse (Negative) reaction or perception related to the stimuli and 9 was the best one (Positive). They were instructed to follow the instruction: "Please, rate your experience after each sound on a scale from 1 to 9; negative experiences will be rate with low numbers and the positives will be with higher values". During the complete procedure, it was measured the lead II of the ECG signal by using the Cyton OpenBCI board [42]. All tests were made between 15.00 and 18.00 h in order to reduce the influence of the circadian cycle in the heart function.

Data processing

The complete procedure to collect data had four different stages: pre-processing, dataset augmentation, feature extraction, and classification; in addition, a feature ranking stage was made. As a first step, the ECG captured signals were pre-processed; in this procedure, the baseline was removed by applying a third-order one-dimensional median filter and after that it was subtracted from the original signal. Pre-processing of ECG signals is carried out to get a clean ECG signal, by reducing the effects of adverse factors such as Gaussian noise, muscle artifacts, power-line interference, and baseline wander, where baseline wander is a noise source with frequency content less than 1.5 Hz [43]. R-peaks were segmented by using the Pan-Tompkins algorithm [44]; undetected peaks were marked manually. This segmentation is carried out to extract the HRV signal [45] through the time difference between R-peaks, by measuring R-distances in milliseconds [46]. Data augmentation was implemented by applying circular shift and hyperspectral data augmentation; these procedures were applied with the methodology described in [47] and [48, 49], respectively. As results of these processes, the order of the RR intervals is changed, and noise is introduced in the principal components of the data. The dataset original had 780 instances and after the data augmentation process was incremented to 7020. Data augmentation is a very useful technique to generate more samples from which algorithms can learn improving their accuracy, as well as overfitting can be reduced and generalization increased [[50], [51]].

Feature extraction and reduction of dimensionality

After the pre-processing and data augmentation stages, a feature extraction process was realized from each signal segment. In the feature extraction process, temporal, frequential and non-linear domain features were considered from the HRV. The extracted features represent in a compressed form the HRV data, depicting behaviours or patterns from different domains such as time and frequency (Table 2). Details about HRV features could be found in this reference [52]. In the analysis, independently of their physiological interpretation, selected features are considered as descriptors of ECG signals. After the feature extraction, in order to reduce dimensionality in the extracted features, it was done a ranking of the best features with the scoring method Information Gain Ratio [53]; from the ranking process, the 11 best-ranked features were chosen to apply the classification process. The number 11 was selected with respect to a classification analysis with the best features (See below: Section 3.2.2. Harmonic intervals - Classification with the best-ranked features: HMI classes, 24 harmonic intervals). Reduction of dimensionality is carried out to suppress redundant or irrelevant variables; this helps to improve prediction accuracy and reduce the computational cost in training processes, as well as allows a better understanding of data [54].

Table 2

Complete set of HRV features.

HRV features
Higuchi fractal dimension (k = 2)
Higuchi fractal dimension (k = 3)
Higuchi fractal dimension (k = 4)
Hurst exponent
Katz fractal dimension
HRV detrended fluctuation analysis alpha 1
HRV detrended fluctuation analysis alpha 2
Mean of the heart rate
Mean R-R interval
Performing triangular interpolation
Triangular index from the interval histogram
Root mean square of the successive differences
Minor semi-axes of the ellipse fitted in the Poincaré plot (SD1)
Major semi-axes of the ellipse fitted in the Poincaré plot (SD2)
Ratio between the axis of the ellipse fitted in the Poincaré plot (SD1/SD2)
Correlation dimension
Approximate entropy
Low-frequency components
Power of low-frequency components
Power of high-frequency components
Ratio of low and high-frequency components
High-frequency components
Very low-frequency components
Total power
Probability of intervals greater 50 ms
Interquartile range of Euclidean distance
Median of Euclidean distance

Classification and evaluation

This research considered two machine learning algorithms to carry out the classification tasks: k-nearest neighbours and Random forest (Table 3). The configuration parameters were chosen by experimentation.

Table 3

Configuration parameters of the classification algorithms.

Classification algorithms	Configuration
k-nearest neighbours	Number of neighbours: 20, metric: Manhattan, weight: distance
Random forest	Number of trees: 45

The training and evaluation process of the model was done through the cross-validation, by considering ten folds. Cross-validation was applied based on the subjects; this procedure was applied five times and the mean of these outcomes was reported. This method reduces the randomness from splitting the data only once, reduces the overfitting, and increases the replicability of the outcomes [55]. Additionally, the Matthews Correlation Coefficient (MCC), sensitivity, specificity, accuracy, and – AUC – area under the ROC curve (receiver operating characteristics) were taken as assessing metrics. The MCC is a metric to assess the performance of predictors or classifiers and it is very useful since it could be used even with imbalanced data; a value of 1 means an ideal prediction, -1 represents a total inverse prediction, and 0 is related to random processes [56]. In this research, the link between HRV and the administered stimuli was done throughout the classification of several features extracted from HRV signals. In this case, considering the experimental design, a good classifier performance, represented by high scores in the assessing metrics, can be associated with a relation cause-effect between the stimuli and heart response, measured by HRV signals.

Results

This section presents the results of two ways of analysis. The first one studies the response of the heart to two different stimuli, i.e., noise and harmonic intervals; the second analysis shows a deeper view by considering the different types of noises and harmonic intervals separately.

Noise and harmonic music intervals

As a first analysis, it was performed a classification of the stimulus types, i.e. noise, and harmonic intervals from sets of features of the HRV signal and using four different classifiers (Figure 1). From the outcomes is possible to see that in general, the classification algorithms are able to differentiate between the classes of noise and HMI. The performance in this classification was equal to or higher than 0.84 in all metrics considered. MCC and AUC are very important in this classification since this is carried out on an imbalanced dataset.

Figure 1

Classification performance of the type of classes (stimuli): noise and harmonic intervals (Table 1).

Noises and harmonic intervals as independent classes

Once the study with two classes ended, to take a closer observation into the stimuli of noise and harmonic intervals, it was done an analysis with one type of class at once, i.e. noise and harmonic intervals separately.

Noise

The first analysis with one type of sound was done with the noise class. It was realized with the Frequency Class of Table 1 (Figure 2) and by classifying the six different types of noise used in this research (Figure 3). The performance in classification tasks with noise was equal to or higher than 0.85 in all metrics considered. In these classifications is observed that specificity is higher than sensitivity which suggests that the algorithms have more probability to detect true negatives than true positives (Figures 2 and 3).

Figure 2

Classification performance of the noise classes: low, high, and low-high band frequencies (Table 1).

Figure 3

Classification performance of the noise classes: Blue, Brown, Grey, Pink, Purple and White noise (Table 1).

Harmonic intervals

After the analysis of noise, the harmonic intervals in three different aspects were studied (Table 1): low and high frequency (Figure 4), i.e. octaves 2 and 4 respectively, consonant and dissonant (Figure 5), and each interval separately (24 classes, Figures 6 and 7). The performance in classification tasks with harmonic intervals was equal to or higher than 0.80 in all metrics considered. Unlike previous results - Figures 2 and 3 - the outcomes in Figures 4 and 5 show similar levels in sensitivity and specificity, i.e. the similar probability to detect true negatives than true positives. Contrary to this, the results in Figure 6 are similar to Figures 2 and 3, where detection of true negatives has more probability than true positives.

Figure 4

Classification performance of the HMI classes, octaves 2 and 4 with harmonic intervals (Table 1).

Figure 5

Classification performance of the HMI classes, consonance and dissonance (Table 1).

Figure 6

Classification performance of the HMI classes, 24 harmonic intervals (Table 1).

Figure 7

Random forest classification with ranked features: HMI classes, 24 harmonic intervals (Table 1).

Classification with the best-ranked features: HMI classes, 24 harmonic intervals

Within the analysis of each interval separately, classification performance according to the number of the best-ranked features is presented (Figure 7 and Table 4); this analysis was carried out by considering the Random forest classifier (since this was the algorithm with the best general performance in the whole study), and two metrics, accuracy and MCC. This analysis was carried out to observe the impact of using different numbers of features in the classification performance and to show what features might be more affected by the stimuli; in this case HMI.

Table 4

Ranking of HRV features using information gain ratio (IGR).

Ranking	HRV feature	IGR	Ranking	HRV feature	IGR
1	Higuchi fractal dimension (k = 4)	0.0340	14	Ratio of low and high-frequency components	0.0214
2	Performing triangular interpolation	0.0315	15	root mean square of the successive differences	0.0210
3	High-frequency components	0.0312	16	Katz fractal dimension	0.0209
4	Major semi-axes of the ellipse fitted in the Poincaré plot (SD2)	0.0270	17	triangular index from the interval histogram	0.0207
5	Total power	0.0269	18	Interquartile range of Euclidean distance	0.0204
6	Low-frequency components	0.0262	19	Higuchi fractal dimension (k = 3)	0.0194
7	Probability of intervals greater 50 ms	0.0256	20	correlation dimension	0.0192
8	Median of Euclidean distance	0.0248	21	Hurst exponent	0.0184
9	Ratio between the axis of the ellipse fitted in the Poincaré plot (SD1/SD2)	0.0241	22	approximate entropy	0.0171
10	Higuchi fractal dimension (k = 2)	0.0238	23	Mean of the heart rate	0.0135
11	Minor semi-axes of the ellipse fitted in the Poincaré plot (SD1)	0.0222	24	mean R-R interval	0.0133
12	Power of low-frequency components	0.0214	25	HRV detrended fluctuation analysis alpha 1	0.0094
13	Power of high-frequency components	0.0214	26	very low-frequency components	0.0059
			27	HRV detrended fluctuation analysis alpha 2	0.0045

Perception analysis

In addition to the discrimination between the different types of sounds, the capacity to separate the sound according to the perception of subjects was also studied. Both ignoring the class of sound when considering the sound type (noise and HMI, Figure 8). This analysis considered the classes listed in the section 2.1. Experimental procedure - Sounds. Since these classification tasks were carried out on unbalanced datasets, the performance evaluation is presented based on the Matthews correlation coefficient (MCC) for all of the classifiers. These classifications were accomplished with 3 and 9 classes of perception, on the different considered stimuli - Noise and HMI, Noise, and HMI independently. The MCC for classification performance was equal to or higher than 0.80. The best performance was achieved by the Random forest classifier, where the highest MCC value – 0.94 - was achieved with noise stimuli and the scale with 9 values for perception.

Figure 8

Matthews correlation coefficient of the sound perception classification with three and nine in relation to all stimuli (noise and HMI), noise, and HMI (Table 1).

Descriptive statistics of the HRV features

After the classification analysis, descriptive statistics of the HRV features were implemented in order to make observable the changes of the HRV features according to the presented stimuli (Table 1). The Kruskal-Wallis statistic-test was used to determine if there were significant differences between the features [57]; a p-value less or equal to 0.05 was considered to be statistically significant (Table 5).

Table 5

Descriptive statistics of the HRV features according to the presented stimulus.

Feature		Class		p-value
		Noise	Harmonic intervals
Higuchi fractal dimension (k = 2)	Mean ± SD	1.72 ± 0.40	1.64 ± 0.39	0.001
	Median	1.68	1.61
probability of intervals greater 50 ms	Mean ± SD	0.38 ± 0.15	0.35 ± 0.14	0.004
	Median	0.38	0.36
ratio between the axis of the ellipse fitted in the Poincaré plot	Mean ± SD	0.97 ± 0.31	0.92 ± 0.32	0.001
	Median	0.92	0.83
		Harmonic intervals
		Consonant	Dissonant
Higuchi fractal dimension (k = 3)	Mean ± SD	1.91 ± 0.37	1.86 ± 0.33	0.001
	Median	1.87	1.76
ratio between the axis of the ellipse fitted in the Poincaré plot	Mean ± SD	0.94 ± 0.32	0.90 ± 0.30	0.001
	Median	0.85	0.78
ratio of low and high-frequency components	Mean ± SD	0.59 ± 0.29	0.62 ± 0.32	0.022
	Median	0.47	0.55
		Valence
		Negative	Positive
Higuchi fractal dimension (k = 2)	Mean ± SD	1.60 ± 0.36	1.74 ± 0.43	0.001
	Median	1.57	1.68
Higuchi fractal dimension (k = 3)	Mean ± SD	1.84 ± 0.36	1.98 ± 0.43	0.001
	Median	1.79	1.92
mean of the heart rate	Mean ± SD	0.93 ± 0.14	0.97 ± 0.11	0.001
	Median	0.96	0.95

Results of the subjective valence judgements of the musical sounds

In parallel the subject judged the degree they felt the presented sound was more positive or more negative (i.e. in valence). They were asked to express this on a scale of 1 (negative) to 9 (positive).

In order to present sounds that would differ in valence we have constructed harmonic musical intervals different in pitch distance. These intervals have in the musical practice different degrees in consonance or dissonance, which in theory would differ in valence. The intervals were presented in two spectral positions: one low in the range of the male voice and one two octaves higher. Furthermore, some noise bands that differ in the point of gravity of their spectrum were added. The valence of all these stimuli can be compared with the spectrum of the familiar sounds in human speech [58].

As the subjects did not all use the same part of the response scale their values were normalized by subtracting the mean and dividing them by the standard deviation of their judgements. This way we became a judgment matrix of the sounds versus the subjects.

In Figures 9 and 10 the valence judgements are given as summed over all the subjects after normalisation. In Figure 9, the top left subplot (a) presents the ratings over all sounds in three sections, intervals in low octave (index 1 to 12), intervals in high octave (index 13 to 24), and noises (index 25 to 30); the sounds are presented in the same order of that of Table 1. It is very clear that the intervals in the lower octave (index 1 to 12) and the noises (index 25 to 30) are rated positive, and the intervals in the higher octave (index 13 to 24) are rated negative. The first two right singular vectors presented in the bottom left subplot (b) of the SVD of the judgement matrix indicate two different processes: 1. the difference in spectral position of the harmonic intervals, and 2. the dependence on the precise interval. In subplot of the top right (c), it is possible to see that also in the judgements of the valence of the noise bands have a maximum in position 26 and 27 and taper off towards low and high frequency bands. From the first two right singular vectors (bottom right, d), which is nearly the same as the overall judgement it is clear that there is only one process is happening: the spectral position.

Figure 9

Valence judgements of stimuli.

Figure 10

Valence judgements of HMIs.

In Figure 10, the top left subplot (a) shows the judgments of the HMI in the low octave. The indexes also represent the distance in semitones between the two tones of the harmonic intervals. The intervals with 3, 4, 5, 7, 8, and 9 semitones are rated positive, the interval 1, 2, 10, and 11 as negative. The tritone of 6 semitones is rated as 0. This corresponds quite well with the theory on consonance. The machine learning classification (random forest) of heart response to consonance and dissonance intervals in the low octave agreed very well with subject perception (bottom left, b). The top right subplot (c) reveals that the higher valence goes with lower recognition of the consonance intervals. Only the minor third and the fifth jump out. In the high octave, while subject perception was not closer to the consonance theory as in the low octave, the machine learning classification (random forest) of heart response was as similar as the observed in the low octave, following completely this theory (bottom left, d).

Discussion

This research studied the effects of the harmonic intervals in two separate octaves and in addition to some types of noise on the activity of the heart, HRV features. In this case, it was searched a heart response related to specific elements of music; harmonic intervals and, noise sounds were included as a variation in the stimuli. The outcomes showed the heart response after ten seconds of exposition to the stimuli; this duration of stimulus exposition was similar to the reported in the IADS-2 database [59].

In fact, there is an influence of the selected stimuli over the heart behaviour, specifically in some features of the HRV. At this point, it is important to mention that despite most of the HRV analyses have been done in the long-term, some of these features in short-term recordings [60] have also been carried out. In this study, HRV measures were extracted from ECG signals along ten seconds duration with a main purpose to describe the signals in question. In this case, this short-term response in HRV was related to the interpretation in the long-term; however, it is important to clarify that it is required the validation of this association and to study which ultra-short HRV features can be considered as good descriptors [61]. This is an exploratory study to determine if there is a heart response - HRV - to harmonic musical intervals and coloured noise by means of two different algorithms of classification: kNN and Random forest.

The results also showed that it is possible to discriminate with high accuracy the heart response to two different types of stimuli: noise and harmonic intervals. It is also possible to infer that the heart behaviour connected to these stimuli has a complex nature, so it is necessary to take several features in order to classify the response with MCC higher than 0.84 (Figure 1). As a consequence, this is a multidimensional task. The results suggested the heart behaviour of the subjects was influenced in a different way by the different types of sounds used in the experiment. With a descriptive statistical analysis, it was possible to observe that Higuchi fractal dimension (k = 2), probability of intervals greater 50 ms, and ratio of standard deviation 1 and standard deviation 2 of the Poincaré plot had higher values in the condition of noise than in the condition of HMI (p < 0.005). It is observed that unordered sounds (noise) produced different responses in comparison with ordered sounds (HMI); noise incremented the fractality respect to HMI. It would be interesting to determine in future research if this behaviour is reproducible with other types of ordered and unordered sounds.

After the analysis of two classes, a deeper examination was performed with one class of sound, i.e. the noises and the harmonic intervals were studied individually. First, in the noise class, the response to the sound was discriminated in relation to its frequency content (Table 2), namely low, high, and low-high bands (all frequency bands) and regarding the noise types. In both cases, an MCC above 0.85 was achieved (Figures 2 and 3). These results meant the heart behaviour changed with each noise of diverse frequential content or type; i.e. different types of noise, according to their frequency bands, produced distinct effects on the heart.

As a second part of the examination, with one class of sound, the Harmonic music intervals (HMI) were analysed. Heart responses were classified regarding stimulus characteristics such as frequency content – low and high octave -, consonance – consonance and dissonance -, and as individual sounds – 24 HMI - (Figures 4 and 5, and 6). Again, in all cases, an MCC superior to 0.80 was achieved. The success of these classification processes suggests the heart response was affected in a different way by the octave of the sounds (high or low frequency), their consonant or dissonant nature, and by each harmonic interval sound independently.

The feature ranking in the process of classification of each interval separately revealed the extent of the contribution to this task of each feature in turn (Table 4). As might be expected, the very low-frequency components made no contribution because of the duration of the HRV record/analysis (10 s); additionally, it is important to note how the mean of the heart rate and the mean R-R interval also each contributed little to this classification. Bearing this in mind, it would be possible then to say that the HMI stimuli did not produce changes in the heart rate and the mean R-R interval. The first four or five features meanwhile made the biggest contribution to the classification process, since with these it was possible to achieve metrics of accuracy and MCC higher than 0.8. In this light, features such as Higuchi fractal dimension (k = 4), high-frequency components, and total power made a substantial contribution to this discrimination task; it would thus be fair to state that the HMI stimuli produced bigger changes in these features.

The analysis with descriptive statistics (Table 5) showed in this case that Higuchi fractal dimension (k = 3) changed from 1.86 for dissonant to 1.91 for consonant intervals (p = 0.001). Unlike than noise stimuli, ordered sounds – consonant - incremented the fractality respect to less ordered sounds - dissonant. The ratio between the axis of the ellipse fitted in the Poincaré plot varied from 0.90 for dissonant to 0.94 for consonant intervals (p = 0.001). The ratio of low and high-frequency components decreased its values from 0.62 to 0.59 for dissonant and consonant intervals (p = 0.022), increasing the parasympathetic dominance [45] (assuming the validity of this ratio for short-term HRV). Finally, regarding of heart response to valence perception, the Higuchi fractal dimension (k = 2) and mean of the heart rate increased from 1.60 to 1.74, and 0.93 to 0.97, respectively. These results might be a possible inspiration for future research in such a way specific sounds such as harmonic music intervals can be used to produced controlled changes on HRV and heart response.

Some aspects of the performance metrics are worth being mentioned. Regarding sensitivity and specificity, in the case of different types of classes/stimuli, or unbalanced datasets, sensitivity was greater than specificity (Figures 1 and 5). In the case of discrimination of classes belonging to the same type, sensitivity was equal to or less than specificity (Figures 2, 3, 4, and 6). In the classification of 24 harmonic intervals (Figure 6), due to the number of classes in this task, specificity tends to take greater values than in problems with few classes; in this case, due to the fact that the algorithms are dealing with a balanced dataset, accuracy provides a better metric of performance. With respect to AUC, this was higher in Random forest than kNN (Figures 1, 2, 3, 4, 5, and 6). MCC was in general higher in kNN in comparison with Random forest (Figures 1, 2, 3, and 4); MCC was higher in tasks directly related to HMI, as it is the case of classification of "consonance and dissonance" (Figure 5), and "24 harmonic intervals" (Figure 6).

The heart response was also analysed (Figure 8) to the sounds regarding the perception of subjects; this procedure was done by including all the types of sounds, i.e. noise and HMI. In this procedure, both kNN and Random forest were able to predict the subject perception both in a general way (with groups of three classes, that could be grouped as postivie, neutral and negative) and, in a more specific manner (with groups of nine classes). In this last case, it was observed that the perception has considerable relevance in the heart response since MCC>=0.8 was achieved in all classifications considering or not the type of the stimuli. Thus, it could be possible to infer that the perception of sound has also an important influence on the effect of the sound in the heart behaviour, and, until a certain point, it could be independent of the type of noise or harmonic intervals.

In respect of the experimental design, the HMI stimuli had the same lower tones, 110 and 440 Hz, both in the lower and in the higher octaves. This fact introduced those notes as a reference for the ear; where a modal or even tonal perception could be introduced in the listeners. I.e. a tonality around the root A - A2 and A4 - could be perceived by the subjects. This general condition could have introduced a bias in the outcomes into the heart response to these HMI. For this reason, it is important to include HMI with different lower notes in future studies. It would be also interesting to determine if the observed heart responses are also observed in HMI stimuli in different octaves than those included in our experimental design, i.e. lower and higher than 2 and 4.

The judgements of the subjects confirmed the relation between valence and consonance (Figure 10). The intervals in the higher octave were judged as less positive indicating the relation with the range of the human voice. The analysis of the heart signals revealed that aspects such as type of sound, frequency content, consonance condition (for HMI), and subjective perception had influence in the heart response to the sound stimuli. Each type of noise and harmonic interval itself originated a distinguishable reaction in the heart. It was possible to recognise the valence judgements by the heart response. While the classification of consonant/dissonant HMI matched with subject judgments, consonant related to positive and dissonant associated with negative.

The subjective perception of subjects agreed closely with the "actual" consonance/dissonance quality of the HMI stimuli in the lower and higher octaves (Figure 10), i.e. subjective perception did indeed have an influence on heart activity. However, it is important to remark that heart activity (HRV/ANS) was also affected not by subjective perception but by physical features of sound, specifically consonance and dissonance characteristics. In this case, there was no direct interaction between physical features of sound and the subjective perception of sound, in other words the heart reacted to consonant sounds similarly to the way in which it might react to other consonant sounds, independent of whether these were subjectively perceived as positive or pleasant. Despite the fact that these findings remain to be confirmed and expanded in future research, they anticipate a promissory tool with which to influence heart activity objectively, so that on determined occasions the subjective perception of listeners might well be discounted in order to standardize procedures or protocols concerned with affecting the heart with sound.

With this research was learned that considered sound had an influence on heart behaviour. Heart responses agreed with the subject judgements; this was very observable in the low octave of HMI sounds. Positive judgments were associated with heart response to consonant sounds and negative judgments to dissonant. An association of heart response with the frequency content of stimuli was observed. In addition to the better agreement between subject judgments and consonance/dissonance quality in the lower octave, algorithms found distinguishable heart responses between low and high HMI. Heart responses were also distinguishable by the algorithms between low, high, and low-high band frequencies of the noise stimuli. Since the heart might be influence by the consonant quality of HMI, and this response agrees with subject perception (valence), this research supports the theory related to the biological influence in the perception of HMI as consonant or dissonant [58]. It is important to note the fact that the noise stimuli were judged in the same range as the stimuli in the low octave. This represents a strong argument for the biological basis of valence.

From the outcomes, it is possible to observe that heart response to sound stimuli was affected by several elements implied. First, the type or nature of the sound produced different responses in the heart (Figure 1). Second, the frequency content of the sounds generated distinct heart reactions (Figures 2 and 4). Here there is an interesting element to study in future works, where it is important to determine if the heart response to frequency content depends or not of the sound type. Third, each sound in itself was able to produce particular reactions on the heart (Figures 3 and 6), where a better distinction in such reactions was noted with the noise sounds (Figure 3). Fourth, in the particular case of harmonic music intervals, their grade of consonance/dissonance contributed to the changes in the heart (Figure 5). Sixth, the experimented perception of subjects also contributed to heart reactions (Figure 8). Thus, heart response to sound stimuli was influenced by these six factors; a great capability of affectation and sensibility of heart to stimulus characteristics and perception of subjects was observed. Bearing this outcome in mind, in future research it is interesting to discover new factors that might influence the heart reaction to sound stimuli.

Conclusions

In this research, it was possible to establish differences between the heart response to sound noises and harmonic intervals by using tools of machine learning. With these tools was possible to determine that HRV features had the ability to represent the heart response to the selected stimuli. Aspects such as type of sound, frequency content, consonance condition (for HMI), and subjective perception had influence in the heart response to the sound stimuli. Thus, each type of noise and harmonic interval itself originated a distinguishable reaction in the heart. In the particular case of the harmonic intervals, it is interesting to note how the effect of consonance quality could be also found in the heart response. This study found support for a heart response to harmonic music intervals and coloured noise beyond the conscious processing of the subject. This fact involves a biological basis of valence and the perception of HMI as consonant or dissonant. This study represents a substantial basis for music therapy and suggests the development of new studies to establish a new solid basis in regards to the effects that elemental parts of music could produce on the human body.

Declarations

Author contribution statement

Ennio Idrobo-Ávila, Rubiel Vargas-Cañas: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Humberto Loaiza-Correa, Flavio Muñoz-Bolaños, Leon van Noorden: Analyzed and interpreted the data; Wrote the paper.

Funding statement

This work was supported by , , and Colciencias (Funding call No. 727 of 2015), Colombia.

Data availability statement

Data will be made available on request.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.