Predicting the crossmodal correspondences of odors using an electronic nose.

When designing multisensorial experiences, robustly predicting the crossmodal perception of olfactory stimuli is a critical factor. We investigate the possibility of predicting olfactory crossmodal correspondences using the underlying physicochemical features. An electronic nose was tuned to the crossmodal perceptual axis of olfaction and was used to foretell people's crossmodal correspondences between odors and the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, and colors. We found that the underlying physicochemical features of odors could be used to predict people's crossmodal correspondences. The human-machine perceptual dimensions that correlated well are the angularity of shapes (r = 0.71), the smoothness of texture (r = 0.82), pitch (r = 0.70), and the lightness of color (r = 0.59). The human-machine perceptual dimensions that did not correlate well (r < 0.50) are the perceived pleasantness (r = 0.20) and the hue of the color (r = 0.42 & 0.44). All perceptual dimensions except for the perceived pleasantness could be robustly predicted (p-values < 0.0001) including the hue of color. While it is recognized that olfactory perception is strongly shaped by learning and experience, our findings suggest that there is a systematic and predictable link between the physicochemical features of odorous stimuli and crossmodal correspondences. These findings may provide a crucial building block towards the digital transmission of smell and enhancing multisensorial experiences with better designs as well as more engaging, and enriched experiences.

Introduction

Our brain constantly combines information from different sensory modalities to better comprehend our surrounding environment [1]. The integration process influences a person's interpretation and the subjective experience that goes with it [2]. Crossmodal correspondences are the consistent and non-arbitrary associations that occur between stimulus features in different sensory modalities (see [3] for a review). For example, people consistently associate a fruity odor to pink/red colors and a musty smell with brown/orange colors [4]. The main mediation factor behind crossmodal correspondences is presumed to be hedonics; that is these correspondences mainly occur based on how pleasant the inducing stimuli is perceived [5]. These associations can be considered as a sensory expectation; inconsistency with the expected and actual attributes of an experience results in the experience being perceived as less pleasant [6]. Furthermore, tailoring the characteristics of an experience, be it consistent or otherwise, can modulate the user's experience towards the desired outcome. For example, it has been shown that odorless coloring of white wine, so it appears red, can bias the decision of expert wine tasters and the olfactory expert descriptions of the dyed white wine matched those of the red wine [2]. However, when designing multisensorial experiences, the designers often have an incomplete set of known sensory cues, even more so with novel odorants. Generally, designers want to stimulate the sensory modalities harmoniously and consistently to increase the psychological impact [7]. Extensive psychological tests are usually conducted to uncover the sensory expectations of the consumers for a particular experience; these expectations are then often reflected in the experience.

When volatile molecules enter our nasal cavity, our olfactory receptors detect these molecules and the olfactory information is projected via the olfactory bulb to the olfactory cortex [8]. A physical, perceptual, and semantic representation of the odor is formed via a neural signal transmitted in the olfactory pathway [9, 10]. This pathway shares a common neural substrate called the limbic system [11], which deals with mood and emotional processes. Our nasal cavity contains thousands of olfactory receptors [12] that are believed to recognize specific chemical features [13], and olfactory perception is rooted in the chemical properties of volatile molecules [14]. Psychophysical evidence suggests that the pleasantness of odors is encoded in the physicochemical structure of odorous molecules [15]. Like the human olfactory stimulus, electronic noses (e-noses) consist of an array of sensors. Each of the sensors in the e-nose has a limited detection capability, and the specificity comes from the number of sensors. An e-nose can detect specific smells more efficiently than the human nose due to its higher sensitivity and specificity. The basic principle behind e-noses is to convert odors' "chemical footprint" into a series of electrical signals; these signals are often indicated by a change in resistance. E-noses consist of three major components – a sample delivery system, an array of sensors (predominantly gas), and a pattern recognition system for the classification and characterization of an odor source rather than the output of a specific sensor [16, 17]. E-noses are primarily used to discriminate, classify, and occasionally quantify different odorous compounds in the vapor phase [18], such as fungal identification and classification [19], assessing beer quality [20], and assessing the aroma profile of coffee [21]. The first conceptualization of this device came from Persaud and Dodd [22] and was designed to mimic human olfaction.

A few studies have attempted to link the physicochemical features of odors to human perception. Haddad et al. showed that it was possible to connect the series of electrical signals produced by the MOSES II e-nose to the perceived pleasantness of odors. A neural network with a singular hidden layer and five neurons was used to extract the pleasantness of odorants. The input consisted of manually extracted features (i.e., signal max) [23]. Wu et al. improved upon the model initially presented by Haddad et al., by incorporating a non-uniform sampling algorithm as the feature extraction method, adding additional odorants to the dataset, and developing a convolutional neural network for the classification of the perceived pleasantness [24]. Schiffman et al. showed the perceived intensity of odors could be linked to the two different e-noses (NST 3320 and Cyranose® 320) [25]. Burl et al. used an e-nose to predict the perceptual descriptors of odors, for example, "Minty" and "Sour". Their results also revealed that several regression models needed to be developed, as each model was only capable of reliably predicting a few of the descriptors [18]. Overall, these results show the physicochemical features of odors transduced by an e-nose can be linked to different aspects of perception. However, it is still unknown if the physicochemical features of odors can predict crossmodal correspondences.

The focus of this study was to investigate if the physicochemical features of odors could be used to predict crossmodal correspondences. Two different brands of essential oils were used to present chemical diversity to the underlying physicochemical features. Data on the crossmodal perception of odors (the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, and colors) for various essential oils were collected in prior work [5]. E-nose responses were collected for the different odorants using the same brand and volume as the perceptual data. The odors used in the confines of these experiments were chosen as they are commonly used in perfumes and olfaction enhanced multimedia. Robust predictions of the human perception of odors are crucial for constructing an artificial olfactory system, industrial quality control applications, and designing multisensorial experiences where assessments must be made in accordance with human perception.

Methodology and materials

Electronic nose

We used the e-nose initially presented in [16] and modified in [26] to record the different odors. The gas sensors used in the e-nose are as follows: MP503, BME680, MQ3, MQ5, MQ9, and a WSP2110. All sensors were connected to an Arduino MKR1000 microcontroller chosen for its integrated Wi-Fi capabilities. The e-nose was designed and built in the Immersive Reality Laboratory, University of Liverpool. These sensors were selected to detect a wide variety of different odors and not just essential oils; however, emphasis was given towards the detection of essential oils. The e-nose deals with several types of gases by employing cross-reactivity into the design of the sensory array, whereby different gaseous mixtures will produce variable responses. The differentiation of gaseous mixtures is determined by the intensity of all of the features provided by the respective sensors. The e-nose was controlled using custom software. A schematic representation of the e-nose is shown in Figure S1 (Supplementary materials). The sensor data was sent to a nearby computer in real-time using UDP (user datagram protocol). A packet was received every ≈250 ms, resulting in ≈4 packets every second. Any out-of-order packets were placed back in order after being received. Each odor recording consists of 11 features: time, air quality, pollution level, temperature, pressure, humidity, gas, MQ3, MQ5, MQ9, and HCHO; for more information on the sensors, see Table 1. A photo of the e-nose/experimental setup is shown in Figure S2 and a diagram of the e-nose circuit is shown in Figure S3 (Supplementary materials).

Table 1

Gas sensors used in the e-nose with their range (ppm) and detectable gases. It is important to note that the sensors may also respond to gases not included in this table.

Gas Sensor Name	Detection Range (ppm)	Detectable gases	Sensor Output Name
MP503	10–1000	Alcohol & Smoke	Air Quality & Pollution Level
BME680	0–500	IAQ	Temperature, Humidity, Pressure, & Gas
MQ3	0.05–10	Alcohol, Benzine, CH4, Hexane, LPG, & CO	MQ3
MQ5	200–10000	LPG, Natural Gas, Town Gas, Alcohol, & Smoke	MQ5
MQ9	10-1000 CO100-10000 Gas	Carbon Monoxide, Coal Gas, & Liquefied Gas	MQ9
WSP2110	1–50	HCHO, Toluene, Methanol, Benzene, & Alcohol	HCHO

Chemical data

The prepared solutions consisted of 4 mL of the respective essential oil. Five were from Mystic Moments™ (caramel, cherry, coffee, freshly cut grass, and pine) and five from Miaroma™ (black pepper, lavender, lemon, orange, and peppermint). Each solution was placed in the same position for all the recordings to negate distance-based sensor bias. The lid of the e-nose was closed while recording the odors and the e-nose was flushed with ambient air for a minimum of 30 min between recordings. Each recording was 10 min in duration, with one hundred prepared for the experiments; ten recordings were prepared for each of the odors. Before any of the recordings were used in the analysis, they first underwent pre-processing to reduce the dimensionality and to remove noise from the generated signals. The pre-processing involved taking the mean over 1-s intervals creating a 600 × 10 matrix. The signal for each sensor response was then smoothed using a three-point moving average filter. The median value from each sensor for each recording was then used for the analysis; This resulted in a final dataset of 100 × 10. One row for each recording and one column for each of the features from the e-nose, excluding the time component. Figure 1 shows the sensor signal processing pipeline for the air quality feature, albeit this was conducted on all features excluding time.

Figure 1

Sensor signal processing pipeline for the air quality feature of one of the lemon essential oil recordings.

Perceptual data

The perceptual data collection is described in [5]; here, we analyzed a series of olfactory crossmodal correspondences and explored the nature and origin of these associations. We found consistent crossmodal correspondences between odors and the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, colors, emotional, and musical dimensions. Sixty-eight participants were presented with ten unlabeled odors and were asked to associate ratings to the respective perceptual dimensions. Two essential oil brands were used to provide chemical diversity in the underlying data, consequently demonstrating the robustness of the developed models. All odors were presented to the e-nose in the same manner and volume as presented to the participants. Participants were placed in a lightproof anechoic chamber with an overhead luminaire (GLE-M5/32; GTI Graphic Technology Inc., Newburgh, NY). The lighting in the room was kept consistent by using the daylight simulator on the overhead luminaire. All participants provided informed consent before taking part in any experiments. This study had ethical approval from the University of Liverpool and was conducted under the standards set in the Declaration of Helsinki for Medical Research involving Human Subjects. For more information on the presented stimuli, see Sections 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5.

Shape stimuli

A nine-point scale was used to measure the association between a particular odor and the angularity of the visually presented shapes. The endpoint of the scale was an angular shape and a rounded shape on scales right and left side, respectively. The middle of the nine-point scale (5) was neutral (no opinion).

Texture stimuli

Similarly, to measure the association with the smoothness of the odors a nine-point scale was used with the words "rough" and "smooth" on the scales on the right and left side, respectively. Participants were also presented with physical representative textures to help them decide, with silk representing smoothness and sandpaper being a representative for roughness. Participants had to feel the textures at least once during the question's first appearance. The middle of the nine-point scale (5) was neutral (no opinion).

Pleasantness stimuli

A nine-point scale was used to measure the perceived pleasantness of the odors with the words "very unpleasant" and "very pleasant" on the left and right sides, respectively. The middle of the nine-point scale (5) was neutral (no opinion).

Pitch stimuli

The full range of audible frequencies was implemented using a slider, with the left side being 20 Hz and the right side being 20 kHz. Each time the slider was adjusted, the respective frequency was played, producing a sinusoidal tone lasting 1-s in length. Participants were first played samples from either end of the scale and then a sample approximately halfway between the two points. Participants answered higher or lower until they felt like the pitch matched the odor, the slider was adjusted accordingly. A binary search approach was implemented to reduce the time required to complete the question.

Color stimuli

The CIELAB color space, more commonly known as the L∗a∗b∗ color space, was created to be a perceptually uniform color space where the Euclidian distance reflects an approximately perceptual difference. The L∗a∗b∗ color space is expressed in three channels: L∗ (lightness), a∗ (red-green) and b∗ (blue-yellow). See [27] for more information about the L∗a∗b∗ color space. Participants could freely select a color from the L∗a∗b∗ color space that they believed best matched the current odor. A color patch was displayed to the participant after they selected a color. Participants could slide through the L∗a∗b∗ color space by adjusting the lightness (L∗). Only the final color selection was saved.

Data analysis

The analysis was conducted using MATLAB™ 2018b. The odor recordings initially contained eleven features. The time component was not used after the pre-processing stage and was not used in training or testing the regression models. Outliers were removed from the perceptual data, any values outside the range of ±1.5 std of the mean value were replaced with the mean value. Any detected outliers were excluded from the second mean calculation. Before performing a principal component analysis, the underlying data was first standardized using z-score normalization.

Results

Ten odors were recorded using the e-nose; each odor was measured ten times at the same volume (4 mL). The brand of essential oil was selected to align the chemical data to the perceptual data. Example e-nose recordings are shown in Figure 2, which shows how the e-nose signals change over time with no odor Figure 2A and with an odor Figure 2B. Additionally, z-score normalization was conducted on the same data to visualize the trend of the sensor responses with a lesser response over time, for example, temperature. The z-score normalization was conducted on all the sensor values for each sensor separately using the sample standard deviation; four linearly spaced points were then taken and displayed in Figure 2C for no odor and with an odor (Figure 2D). Four points per sensor were displayed to enhance the clarity and interpretation of the data.

Figure 2

Example e-nose recordings over time for (A) no odor and (B) lemon. Z-score normalized sensor responses over time (C) no odor and (D) lemon. Each line represents a different sensor response over time in the e-nose.

Principal component analysis was then conducted on the physicochemical and perceptual data to visualize the similarity between the two. The distance between two points indicates how similar the odors are in their respective space, with closer points indicating a higher degree of similarity. To construct the perceptual dataset, all the ratings were scaled between the range of one to nine. Z-score normalization was then conducted using the sample standard deviation and the grand mean on the perceptual and chemical datasets separately. The chemical dataset's first two principal components explain 75.56% of the total variance, 51.06%, 24.49%, respectively, and is shown in Figure 3A. The perceptual dataset's first two principal components explain 84.39% of the total variance, 46.55%, and 37.83%, respectively, shown in Figure 3B.

Figure 3

Score plots for (A) odors in the chemical space and (B) odors in the perceptual space.

To determine the clusters that achieved similar scores in the chemical and perceptual spaces k-means cluster analysis was conducted on all the principal components. This allowed for the comparison of the similarity of the two spaces on uncompressed versions of the underlying data. We opted for three clusters as the inclusion of four or more would result in an outlier (a singular odor in its own cluster) in at least one of the two spaces. For the chemical space, this revealed that (peppermint and lemon), (coffee and caramel), and (freshly cut grass, lavender, pine, cherry, orange, and black pepper) were chemically similar. In the perceptual space, this revealed that (orange and lemon), (caramel, coffee, and cherry), (freshly cut grass, pine, lavender, peppermint, and black pepper) were perceptually similar. Considering Figure 3A and Figure 3B together, we can see a moderate overlap between the odors in the physicochemical and perceptual space, with coffee, caramel, freshly cut grass, lavender, pine, and black pepper being grouped into similar clusters in either case. This tells us that there is a degree of consistency between these two spaces, and therefore it should be possible to predict people's crossmodal correspondences using the underlying physicochemical features.

To determine which features would contribute to the final models, a multivariate analysis of variance (MANOVA) was first conducted with the e-nose features as the dependent variable and the identity of the odor as the independent variable. This revealed that there is a statistically significant difference in the features (e-nose sensor responses) based on the presented odor (F(81, 538.9) = 37.74, p < 0.0001, R2 = 0.85). To determine which features differ for the presented odor one-way univariate ANOVAs were conducted. This revealed that all features significantly differ (Bonferroni corrected) depending on the odors (all p-values < 0.005): air quality (F(9, 90) = 91.98), pollution level (F(9, 90) = 22.70), temperature (F(9, 90) = 26.90), pressure (F(9, 90) = 65.83), humidity (F(9, 90) = 22.84), gas (F(9, 90) = 29.91), MQ3 (F(9, 90) = 51.93), MQ5 (F(9, 90) = 42.35), MQ9 (F(9,90) = 20.79), and HCHO (F(9,90) = 54.30). These results indicate that all the sensors respond to one or more of the gases in the essential oils and that these responses for each of the features is different depending on the presented essential oil. Therefore, we decided to use all the features when training the regression models.

To determine the optimal regression algorithm to predict people's crossmodal correspondences, we initially trained and tested four different algorithms: linear regression, support vector machine, random forest, and Gaussian process regression (GPR) using fifty-fold cross-validation. We chose these algorithms because they are commonly used in conjunction with the sensor data from e-noses [28, 29, 30, 31]. GPR had the lowest root squared mean error in all cases (see Table S1, Supplementary materials). Therefore, we decided to optimize the kernel function for each of our models by selecting the kernel function that best represented the underlying data (see Table S2, Supplementary materials). We believe that GPR has the lowest error in all cases because it works well with noisy data [29] and small datasets. Pearson correlations were used to calculate the within and between odor correlations. The human-human correlation between the participant's responses and the median ratings was calculated. This revealed correlations of (r = 0.55, p < 0.0001) for the angularity of shapes, (r = 0.35, p < 0.0001) for the smoothness of texture, (r = 0.43, p < 0.0001) for the perceived pleasantness, (r = 0.39, p < 0.0001) for pitch, (r = 0.56, p < 0.0001) for the color dimension L∗, (r = 0.57, p < 0.0001) for the color dimension a∗, and (r = 0.60, p < 0.0001) for the color dimension b∗. To test the quality of our models on unseen odors, we opted for a leave one odor out approach. The models were trained ten times using all the recordings for nine of the odors and all the recordings for one odor for testing. This resulted in a 90 × 10 matrix for training and a 10 × 10 matrix for testing for every iteration; this was repeated ten times until all the odors had been tested in an unseen state. The human-machine correlations were then calculated using the median ratings and machine regression ratings from when the odors were in an unseen state; a slope of one indicates perfect machine performance, and a slope of zero indicates that a relationship does not exist between the predicted v. actual ratings. This revealed significant correlations for the angularity of shapes (r = 0.71, p < 0.0001), the smoothness texture (r = 0.82, p < 0.0001), the perceived pleasantness (r = 0.2, p = 0.0484), pitch (r = 0.70, p < 0.0001), the color dimension L∗ (r = 0.59, p < 0.0001), the color dimension a∗ (r = 0.44, p < 0.0001), and the color dimension b∗ (r = 0.42, p < 0.0001), see Figure 4.

Figure 4

Predicted v. actual plots for (A) the angularity of shapes, (B) the smoothness of texture, (C) the perceived pleasantness, (D) pitch, (E) the color dimension L∗, (F) the color dimension a∗, and (G) the color dimension b∗.

From Figure 4, we can see that for some of the sensory modalities, the machine predictions correlated well with the human ratings (r > 0.50). The sensory modalities that correlated well are the angularity of shapes, the smoothness of texture, pitch, and the lightness of color. The sensory modalities that did not correlate well are the perceived pleasantness and the color dimensions (a∗ and b∗). These findings indicate that the correspondences people have towards odors can be robustly predicted (all p-values < 0.0001 except for the perceived pleasantness) using the underlying physicochemical features, albeit the amount of variance captured for the a∗ and b∗ dimensions is not overwhelming.

To further investigate the statistical robustness of our findings, we randomly shuffled the perceptual data associated with a given odor one hundred times for each perceptual dimension and repeated the regression analysis. For example, the response values for lemon now might be the response values for black pepper. The average prediction rates dropped for all perceptual dimensions to r = -0.16, p = 0.11 for the angularity of shapes, r = -0.04, p = 0.15 for the smoothness of texture, r = -0.18, p = 0.15 for the perceived pleasantness, r = -0.14, p = 0.14 for pitch, r = -0.04, p = 0.23 for the color dimension L∗, r = -0.12, p = 0.20 for a∗, and r = -0.19, p = 0.14 for b∗. These findings reflect the model's ability to predict the crossmodal correspondences of odors rather than the reported correlations being obtained due to internal structure.

To test if our results were significantly impacted by our removal of the perceptual outliers, we repeated the prediction analysis with the inclusion of the perceptual outliers. This resulted in moderate change in the correlation coefficient in the perceived pleasantness from (r = 0.20, p > 0.05) to (r = 0.09, p > 0.05), L∗ from (r = 0.59, p < 0.0001) to (r = 0.46, p < 0.0001), and a∗ from (r = 0.44, p < 0.0001) to (r = 0.35, p = 0.0003). A small change was noticed in the angularity of shapes (r = 0.71, p < 0.0001) to (r = 0.64, p < 0.0001), smoothness of texture from (r = 0.82, p < 0.0001) to (r = 0.79, p < 0.0001), pitch surprisingly increased from (r = 0.70, p < 0.0001) to (r = 0.75, p < 0.0001), and b∗ did not change at all (r = 0.42, p < 0.0001) to (r = 0.42, p < 0.0001). The correlations for the angularity of shapes, the smoothness of texture, pitch, and our color dimensions are still significant with inclusion of the outliers leading us to conclude that the results were only slightly influenced by removal of the outliers, hence suggesting that our analysis is robust.

Discussion

In vision, a predictive property of color is the wavelength of light. In hearing, the frequency of sound is a predictive property of tone. However, in olfaction, it is not currently possible to predict the smell of a molecule using its molecular structure [32]. This is presumably because the dimensionality of olfactory perceptual space is unknown and olfactory stimuli do not vary continuously in stimulus space [32, 33]. Comparatively to vision and audition, predicting olfactory percept requires significantly more information rather than a particular attribute; even then, there is still substantial room for improvement. Here we transduced the physicochemical features of odors using an e-nose, transmitted it to a receiver, and predicted people's crossmodal correspondences towards that odor. This means we could predict the correspondences a person would have towards an odor even if the odors were unseen to the models. Our findings suggest that the crossmodal associations people have towards odors are, at least in part, encapsulated in the physicochemical features and, therefore, can be captured by a machine. Our findings show that it is possible to predict the crossmodal correspondences of odors using the stimuli's physicochemical features.

The ability to predict crossmodal perceptual properties was a critical aspect of this work. It will inevitably lead to more refined multi-sensorial experiences in various disciplines, including product design and human-machine interfaces. These results could be considered as a building block for the digital transmission of smell. In theory, the transmission of odors would require recording an odor to reproduce (e.g., intensity over time), deciphering odor composition, transmitting, and reproducing it at the other end. However, the synthesis of odors is currently technically limited. Thus, deciphering the odor along the perceptual axis and aligning different sensory modalities to match the user's sensory expectations could generate a similar percept, even if the odor(s) or physical configurations at either end of the transmission points are not identical. Interestingly this idea could be used to fine-tune interactive and immersive experiences to suit a particular desired outcome. For example, changing the color of a drink affects our perception of the product, shaping the aroma, taste, or flavor [34]. In other words, it is possible to shape the multisensory perception of products and experiences towards a more favorable or desired outcome, such as enhancing the perceived pleasantness or creating perceptual illusions.

The main perceptual axis of olfactory perception is pleasantness [35] which the e-nose signals could be partially encapsulating [23]. However, the model that was created for the perceived pleasantness did not correlate well, suggesting that if the perceived pleasantness were partially encapsulated in the underlying signals, its contribution would be minimal. It is also possible that the e-nose signals may encapsulate aspects of intensity [25]. The perception of odors is a complex process that involves both learnt and innately tuned components [15]. It is important to emphasize the limitation of our findings; olfactory perception and successive neural representations are modulated or influenced by several different factors, such as expectations [36], context [37], multisensory convergence [38], and is a heavily learned process [39]. However, a portion of olfactory perception is suggested to be innate and hard-wired [15, 23, 40, 41], which our results reflect, as the dominant aspects of olfactory perception will not be reflected in the physicochemical features. For example, it has been shown that newborn babies with no exposure to culture or learning are averse to unpleasant odors [41], and rats are averse to the smell of predators even if they are bred for several generations in a predator-free environment [40].

In our prior work, we predicted the color people associated with odors [26]. Building upon this, we show that in addition to the ability to predict the color associated with odors, it is also possible to predict a variety of different crossmodal correspondences (the angularity of shapes, the smoothness of texture and pitch) using the physicochemical features of the presented stimuli. Thereby demonstrating the ability to predict human percept with a e-nose extends beyond the bounds of crossmodal odor-color correspondences. In this work, we also improved on the models to predict the color associated with odors, by fine-tuning the underlying hyperparameters that best expressed the underlying data.

Future research could include exploring how employing crossmodal correspondences in human-machine interfaces could benefit multimodal setups [42], which could be advantageous for people with disabilities [43]. This would involve designing a human-machine interface that considers multimodal interaction patterns, such as crossmodal correspondences. Furthermore, the integration of color-sound correspondences in a human-machine interface has been shown to enhance user performance [44], thereby demonstrating the need for crossmodally congruent interfaces and the need to limit the bottleneck of conducting extensive human trials before the development of a compatible interface can begin. The work reported in the paper could be improved upon by including a larger sample size of odors along with their crossmodal correspondences to enhance the reliability of the generated models. Additionally, more robust and reliable models can be obtained by developing the modals to suit singular nationalities and cultures, as they are an influential factor towards explaining crossmodal correspondences (i.e., [45, 46]). Finally, although we have shown it is possible to align an e-nose to the crossmodal perceptual axis of olfaction for commonly encountered odors it remains to be investigated if this is still the case with novel odorants.

Conclusion

Using a multivariate analysis of variance coupled with post-hoc univariate analysis of variance revealed that all the sensors in the e-nose responded differently depending on the presented essential oil. Hence, it indicates that there are no redundant sensors or features in the data provided by the e-nose. Using principal component analysis coupled with k-means cluster analysis the similarity between the odors used in our experiments in the physicochemical and perceptual spaces was explored. This revealed a ≈60% overlap between the two spaces in the physical space, which tells us that there is a reasonable degree of consistency between the two spaces, and that a relationship between the physicochemical features and the crossmodal perceptual ratings exists. Different regression algorithms – linear regression, support vector machine, random forest, and Gaussian process regression were trained and tested using fifty-fold cross-validation. Gaussian process regression gave the lowest error in all cases suggesting that out of the tested algorithms, a Gaussian process best captured the relationship between the physicochemical features transduced by an e-nose and their crossmodal ratings. An e-nose was then aligned to the crossmodal perceptual axis of olfaction, revealing that it is possible to predict the crossmodal correspondences of odors using their physicochemical features, even if the odor was unseen to the generated models. The models for the perceived angularity of shapes, smoothness of texture, pitch, and lightness of color achieved good correlations (r ≥ 0.50, p < 0.0001) and could be robustly predicted. Although the hue of color (a∗ and b∗) could be robustly predicted, their correlations were not overwhelming (r < 0.50). The perceived pleasantness in our case could neither be robustly predicted (p = 0.484) nor captured a decent amount of variance (r = 0.20); this shows that a larger sample size of odors is needed to be able to predict the perceived pleasantness [23, 24]. We then proceeded to investigate the statical robustness of our findings by randomizing the perceptual data, revealing that the obtained correlations were attributed to the model's ability to predict the crossmodal correspondences of odors rather than the internal structure of the underlying data. Finally, we tested if removing the outliers affected our findings by retraining the models without outlier removal, leading us to conclude that the removal of the outliers did not significantly influence the results. The novelty of this work is two-fold; first, we have shown it is possible to align an e-nose to the crossmodal perceptual axis of olfaction, thereby demonstrating it is possible to predict the crossmodal correspondences of odors using their underlying physicochemical features. Secondly, computational models for predicting the crossmodal correspondences of odors have been developed, which could enhance multisensorial experiences with more refined capabilities, leading to more engaging/immersive and enriched experiences as well as better designs.

Declarations

Author contribution statement

Ryan J. Ward: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Shammi Rahman: Analyzed and interpreted the data; Wrote the paper.

Sophie Wuerger, Alan Marshall: Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This work was funded by the (EPSRC) under grant number EP/P004016/1. The funder had no role in study design, data collection, and analysis, decision to publish, or preparation of the manuscript.

Data availability statement

The underlying chemical and perceptual data used in this work has been freely made available at: DOI: 10.5281/zenodo.5601514, along with a README file containing the structure of the data.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

The underlying chemical and perceptual data used in this work has been freely made available at [47] along with a README file containing the structure of the data. DOI: 10.5281/zenodo.5601514.

No additional information is available for this paper.