Tracking historical changes in trustworthiness using machine learning analyses of facial cues in paintings.

Social trust is linked to a host of positive societal outcomes, including improved economic performance, lower crime rates and more inclusive institutions. Yet, the origins of trust remain elusive, partly because social trust is difficult to document in time. Building on recent advances in social cognition, we design an algorithm to automatically generate trustworthiness evaluations for the facial action units (smile, eye brows, etc.) of European portraits in large historical databases. Our results show that trustworthiness in portraits increased over the period 1500-2000 paralleling the decline of interpersonal violence and the rise of democratic values observed in Western Europe. Further analyses suggest that this rise of trustworthiness displays is associated with increased living standards.

Introduction

A number of historical observations suggest that social trust rose steadily in Europe from the early modern period onwards: religious tolerance increased, witch hunts abated, honor killings and revenge lost their appeal and intellectual freedom became a central value of modern countries[1,2]. Historians have used a range of cues to document this process: etiquette manuals, registries of friendly societies, or legal changes[1,3,4]. However, quantitative evidence is scarce and progress in the history of mentalities has been limited by the paucity of tools to capture people’s extinct mental life. Quite obviously, we cannot go back in time and ask people to fill out questionnaires or play economic games[5-7] but we still have access to what their minds produced: books, songs, paintings, sculptures, etc. These cultural artefacts are the remnants of people’s past psychologies and can function as cognitive fossils of extinct mentalities and social preferences. Recent work has indeed shown that people’s preferences in various areas of social cognition are reflected in cultural artefacts: Costa and Corazza[8] demonstrated that the people’s preference for friendly-looking faces leads painters to exaggerate “neotenic” features in their portraits (big eyes or round faces). Similarly, Morin[9] has shown that direct-gaze Renaissance portraits are more popular than averted-gaze portraits. Fictions, such as romance novels[10], TV shows[11], epic poems[12] or tragedies[13], are all consistently aligned with humans’ universal interest for information related to mating, commitment and status competition for reviews and discussions, see refs. [14,15]. These shifts in cultural artefacts reveal global changes in mentalities, reflecting the preference of the sitter, the artist and the audience altogether.

Portraits are particularly promising to document and quantify the level of trust over time. Experimental work have revealed that specific facial features, such as a smiling mouth or wider eyes, are consistently recognized as cues of trustworthiness across individuals and cultures[16-21]. In this paper, we capitalize on this large empirical literature to build an algorithm that estimates trustworthiness based on a pre-identified set of facial characteristics[22]. More precisely, we apply recent machine-learning methods to extract quantitative information about the evolution of social cues contained in portraits. The algorithm generates automatic human-like trustworthiness ratings on portraits based on the muscle contractions (facial action units) detected in facial displays using the open software OpenFace[23]. This algorithm was trained on avatars controlled for trustworthiness and optimized using a random forest procedure (see Supplementary Methods for more details). To assess the generalizability of our model, we then tested its validity on four databases of natural faces rated by real participants. We first demonstrated that the algorithm produced trustworthiness ratings that were aligned with those produced by human participants in all four controlled databases. Another validation method would have been to also measure the correlation between the estimated trustworthiness of the historical portraits calculated by our algorithm and the evaluations of the actual participants on these paintings. This other method has the major advantage of providing a direct test of the reliability of our algorithm. However, since participant evaluations may be influenced by historical cues not relevant to trustworthiness (such as the sitter’s outfit or the painting style) that may bias these evaluations so that older portraits are perceived as less trustworthy, this method of validation is limited. Therefore, we chose to assess the validity and generalizability of our model independently of idiosyncratic biases of participants by relying on well-known effects in the literature, i.e., the effect of emotion, age, gender, and head orientation on facial evaluations.

We thus checked that the algorithm was susceptible to the same biases as humans, i.e., rating younger, feminine, and happy faces as more trustworthy. Third, we checked that the output of the algorithm was robust to variations in head orientation[21,24] (see Supplementary Methods for the results). We then replicated all these findings outside well-controlled databases by analyzing all the images (photographs and paintings) obtained from a Google image search for ‘women portraits’ vs ‘male portraits’ (N = 633; trustworthiness: t(632) = 7.89, p < 0.001; dominance: t(632) = −11.79, p < 0.001). This validation method provides evidence of the ability of our algorithm to produce human-like face evaluations on a large range of images (i.e., controlled photographs, natural photographs and paintings).

Results

Trustworthiness displays in portraits increased throughout history

To assess the evolution of trustworthiness displays in history, we first analyzed the paintings of the National Portrait Gallery (Fig. 1a), the largest online database of historical portraits (analyzed N = 1962 English portraits from 1505 to 2016). Because perceived trustworthiness is correlated with perceived dominance[24], all the analyses were controlled for dominance. In line with historical work, we found a significant increase of trustworthiness displays with time (b = 0.14 ± 0.02, z = 7.49, p < 0.001; Table 1; time coded such as one unit corresponds to 100 years, ±corresponds to standard errors to the mean; Figs. 1b and 2a), suggesting that the value of interpersonal trust increased from the 16th to the 20th century. We then replicated our findings on the Web Gallery of Art, an important fine art repository (N = 4106 portraits) spanning 19 Western European countries seven centuries (1360–1918) and found a significant increase in trustworthiness displays with time (b = 0.07 ± 0.01, z = 5.33, p < 0.001; Table 1; Fig. 2b). Overall, these results are consistent with more qualitative works documenting a so-called ‘Smile Revolution’[25] and a rise of prosocial displays in paintings and in novels[26]. It is worth noting, however, that the historical increase in trustworthiness observed in our datasets parallels the rise of liberal values such as religious tolerance, political freedom and democracy[2,27,28].

Fig. 1

Evolution of trustworthiness displays in England across time.

a Example of faces detected in portraits from the National Portrait Gallery and estimated as lowly trustworthy (top; Thomas Cranmer by Gerlach Flicke, 1545-1546, NPG 535 All rights reserved © National Portrait Gallery, London) and highly trustworthy (bottom; Sir Matthew Wood by Arthur William Devis, 1815-1816, NPG 1481 All rights reserved © National Portrait Gallery). b Evolution of displays of trustworthiness in the National Portrait Gallery (modeled trustworthiness value adjusted for dominance) and GDP per capita in England. Source data are provided as raw data and scripts on the online depository.

Table 1

Effect of time, GDP per capita and democratization on the portraits of National Portrait Gallery and the Web Gallery of Art.

Time only			Affluence only		Time + Affluence		Democratization only		Time + Democratization
	National Portraits Gallery	Web Gallery of Art	National Portraits Gallery	Web Gallery of Art	National Portraits Gallery	Web Gallery of Art	National Portraits Gallery	Web Gallery of Art	National Portraits Gallery	Web Gallery of Art
Year	0.14 ± 0.02 z = 7.49 p < 0.001	0.07 ± 0.01 z = 5.33 p < 0.001			0.08 ± 0.03 z = 3.17 p = 0.002	0.06 ± 0.02 z = 2.87 p = 0.007			0.32 ± 0.11 z = 2.86 p = 0.004	−0.13 ± 0.14 z = −0.98 p > 0.250
GDP per capita			0.03 ± 0.00 z = 7.13 p < 0.001	0.09 ± 0.03 z = 3.16 p = 0.002	0.02 ± 0.01 z = 3.16 p = 0.002	0.07 ± 0.04 z = 1.98 p = 0.048
Democracy index							0.03 ± 0.01 z = 5.24 p < 0.001	−0.01 ± 0.01 z = −1.96 p = 0.051	−0.01 ± 0.01 z = −0.50 p > 0.250	−0.01 ± 0.01 z = −0.96 p > 0.250
Dominance	−0.79 ± 0.02 z = −40.74 p < 0.001	−0.74 ± 0.01 z = −56.58 p < 0.001	−0.78 ± 0.02 z = −40.10 p < 0.001	−0.75 ± 0.02 z = −46.29 p < 0.001	−0.78 ± 0.02 z = −40.30 p < 0.001	−0.74 ± 0.02 z = −46.05 p < 0.001	−0.77 ± 0.03 z = −30.76 p < 0.001	−0.71 ± 0.04 z = 20.17 p < 0.001	−0.77 ± 0.03 z = −30.83 p < 0.001	−0.71 ± 0.04 z = −20.17 p < 0.001
Gender	0.32 ± 0.06 z = 5.64 p < 0.001	0.31 ± 0.03 z = 10.76 p < 0.001	0.29 ± 0.06 z = 5.01 p < 0.001	0.30 ± 0.04 z = 8.31 p < 0.001	0.30 ± 0.06 z = 5.10 p < 0.001	0.29 ± 0.04 z = 7.98 p < 0.001	0.28 ± 0.08 z = 3.61 p < 0.001	0.25 ± 0.07 z = 3.30 p = 0.001	0.25 ± 0.08 z = 3.16 p = 0.002	0.25 ± 0.07 z = 3.37 p < 0.001
Age	−0.00 ± 0.00 z = −2.03 p = 0.043		−0.00 ± 0.00 z = −1.88 p = 0.060		−0.00 ± 0.00 z = −2.26 p = 0.024		0.00 ± 0.00 z = 0.48 p > 0.250		−0.00 ± 0.00 z = −0.17 p > 0.250
Sample (N)	1962	4106	1943	2706	1943	2706	1115	565	1115	565

The first line corresponds to the regression coefficient with their associated standard error to the mean (mean ± s.e.m.). Results in bold corresponds to statistically significant effects of the variables of interest. The upper part of the table presents the effects of the variables of interest (time, affluence and democratization), while the lower part presents the effects of the control variables (dominance, gender and age). All the tests are two-sided. Following APA’s recommendations, exact p-values are provided for p-values between 0.001 and 0.250. Source data are provided as raw data and scripts on the online depository.

Fig. 2

Effect of time and affluence on trustworthiness displays across time.

Time was associated with an increase of trustworthiness displays in both the National Portrait Gallery (a) and the Web Gallery of Art (b)—data are aggregated by decades). Increased GDP per capita predicted increased trustworthiness displays better than time only-models both in the National Portrait Gallery (c) and the Web Gallery of Art (d). Data are represented as mean values, error bars represent standard error to the means, the red line corresponds to the estimated effect in the regression adjusting for gender, age (for the National Portrait Gallery only) and dominance, the shaded area represents the standard error to the mean of these effects. Source data are provided as raw data and scripts on the online depository.

0.07 ± 0.01  = 5.33

< 0.001

= 7.13

< 0.001

0.09 ± 0.03  = 3.16

= 0.002

0.07 ± 0.04  = 1.98

= 0.048

0.03 ± 0.01  = 5.24

< 0.001

Whether such increased trustworthiness in portraits parallels an actual shift in social trust remains an open question. To assess the validity of this assumption, we applied our algorithm to selfies posted on Instagram in six cities around the world in 2013 (Bangkok, Berlin, London, Moscow, New York and Sao Paulo; SelfieCity database, pictured analyzed N = 2277[29]), we found that people located in places where interpersonal trust and cooperation are higher (as assessed in the European and World Value Surveys[30,31]) displayed higher levels of trustworthiness in their selfies (cooperation level: b = 0.13 ± 0.03, z = 3.67, p < 0.001; trust level: b = 0.81 ± 0.23, z = 3.50, p < 0.001; ±corresponds to standard errors to the mean; Supplementary Figure 6). Together, this suggests that the display of trustworthiness in portraits can indeed be used as a reliable proxy of the level of social trust in individuals’ environment[32,33].

Trustworthiness displays in portraits increased with affluence

Another open question is that of the potential predictors of trustworthiness fluctuations in social displays. We first examined the role of resources. Trust can indeed be construed as an investment in social interactions with potential benefits (in the event of cooperation) and also potential losses (in the event of defection). Because losses have more dramatic effects for poorer individuals, individuals with lower resources are arguably more exposed by exploitation risks and should therefore have lower levels of social trust[34]. In line with this reasoning, international surveys show a strong association between resources and social trust[35-38]. Moving beyond correlations, economists have recently demonstrated that childhood resources had a causal impact on adult trust levels using exogenous variations in caloric rationing in post WW2 Germany[39].

This is particularly relevant in light of the fact that the Middle Ages and the early Modern Period were periods of prolonged economic growth for Europe in general and England in particular[40,41]. We thus tested whether higher GDP per capita was associated with the rise of trustworthiness in portraits. Our analysis of the National Portraits Gallery database revealed an association between higher levels of affluence and higher levels of trustworthiness displays between the 16th and the 21st centuries (b = 0.03 ± 0.01, z = 7.13, p < 0.001; Table 1; Fig. 2c), even after adjusting for a monotonous effect of time (b = 0.02 ± 0.01, z = 3.16, p = 0.002; Table 1). Crucially, GDP per capita accounted for the evolution of trustworthiness displays better than a monotonous effect of time (Bayes Factor: 3.38), which suggests that the observed evolution of trustworthiness displays cannot be reduced to a simple cultural accumulation that would have led to the development of painting techniques making sitters look more trustworthy. We then sought to replicate this result in the Web Gallery of Art database and also found a significant positive association between GDP per capita and trustworthiness displays (b = 0.09 ± 0.03, z = 3.16, p = 0.002; Table 1; Fig. 2d). This association was robust to adjusting for a monotonous increase of trust displays over time (b = 0.07 ± 0.04, z = 1.98, p = 0.048; Table 1). Again, the model including GDP per capita provided a better account of the variations of trust displays than time alone (Bayes Factor: 130.16).

Institutional change is another possible predictor of increased trust. The establishment of more democratic, more inclusive and more egalitarian institutions might indeed have created a climate of trust and tolerance[42,43]. We tested this idea by measuring the association between displays of trustworthiness in paintings and political democratization using the Polity2 index (a composite measure of institutionalized democracy and autocracy available from 1800, see Supplementary Methods). Although a significant association was found between these two variables in the National Portraits Gallery (b = 0.03 ± 0.01 z = 5.24, p < 0.001), this effect was not robust to the inclusion of time as covariate (b = −0.01 ± 0.01, z = −0.50, p > 0.250) and the evolution of trustworthiness displays was better explained by GDP per capita than by changes in the institutions (Bayes Factor: 2.75). Moreover, the positive association between more democratic institutions and higher trustworthiness displays was not replicated in the Web Gallery of Art sample (b = −0.01 ± 0.01 z = −1.96, p = 0.051; with time as a covariate: b = −0.01 ± 0.01 z = −0.96, p > 0.250; Bayes Factor of the GDP per capita model compared to the democratic institutions model: 6.16).

Changes in affluence precede changes in trustworthiness displays in portraits

Demonstrating that the association between GDP and the rise of trustworthiness is causal would of course require additional data. Based on our dataset however, we were able to investigate the dynamics of these historical changes by running time-lag analyses on trustworthiness displays and GDP per capita. We found that changes in GDP per capita predicted future changes in trustworthiness displays in the National Portraits Gallery two decades later (F(40,1) = 12.38, p = 0.001) while changes in political institutions did not (F(15,1) = 0.11, p > 0.250). The effect of GDP per capita on trustworthiness displays was generalizable to the other European countries (Web Gallery of Art sample, effect of GDP 20 years before on trustworthiness displays: X(1) = 6.42, p = 0.011; Institutions 20 years before: X(1) = 0.81, p > 0.250). Importantly, changes in trustworthiness displays did not predict future changes in GDP per capita either in the National Portraits Gallery sample (F(41,1) = 0.76, p > 0.250) or in the Web Gallery of Art dataset (X(1) = 2.02, p = 0.155), which suggests that changes in GDP per capita may have preceded changes in trustworthiness displays in this dataset. This conclusion is consistent with other works emphasizing the importance of economic growth and psychological changes in history[44-46].

Discussion

To conclude, our analyses—replicated across two independent fine arts databases—reveals that trustworthiness displays increased in early modern period portraits and are suggestive of an actual shift in social trust over the period (although differences across countries might have persisted over the period, see refs. [5-7]). This cultural shift is more strongly associated with GDP per capita than institutional change. These findings complement existing qualitative historical accounts and demonstrate how insights from cognitive sciences can enrich our understanding of cultural evolution.

Methods

Construction of an algorithm for modeling trustworthiness and dominance evaluations

We built a model that automatically extracts trustworthiness and dominance evaluations from the all the facial action units detected by the OpenFace algorithm (i.e., both dichotomous and continuous estimations; OpenFace version 1.01 using OpenCV 3.3.0[47]). To do so, we extracted the facial action units of five sets of avatars previously generated with Facegen and controlled for dominance, for trustworthiness or for both (Supplementary Fig. 1)[48]. Each avatar is generated from an initial face and manipulated to either express a specific level of dominance, trustworthiness or both based on the model developed by Oosterhof and Todorov[24]. These avatar faces have been shown to successfully elicit ratings of dominance and trustworthiness in participants[48-50]. Thus, compared to participants’ ratings on photographs that may be sensitive to the participants characteristics and to experimental protocol factors (such as the type of scale used to give the ratings), using avatars allow us to have well-validated sets of faces to train our model. These sets of avatars correspond to all the existing and available validated avatars controlled for trustworthiness or dominance and generated by Facegen.

3% of the faces were excluded from the modeling process for not having been accurately detected by OpenFace. The total sample of avatar faces were then split in a training sample (80% of the faces) and a test sample (20% of the faces). Importantly, the percentage of avatars coming from each avatar set was equal in the training and test samples for both trustworthiness and dominance (Trustworthiness: X2(2) = 0.02, p > 0.250; Dominance: X2(2) = 0.01, p > 0.250).

To determine which type of algorithm (linear model, random forest model from the RandomForest R package[51]—Breiman’s random forest algorithm[52]—or support vector model either linear or radial from the kernlab R package[53]) would provide the most accurate evaluations, we ran a repeated 20-folds cross-validation (five repetitions) on the training test of each of these models separately for dominance and trustworthiness using caret R package[54]. Each model’s hyperparameters were optimized using a random search. The hyperparameters optimized for each model are presented in Supplementary Table 1. This analysis revealed significantly better performance for the random forest model than for the linear model and the linear SVM model in terms of mean absolute error, root square mean error and r-squared and was and better than, for the trustworthiness model, and similar to, for the dominance model, the radial SVM model (Supplementary Table 1). For both trustworthiness and dominance, the optimal mtry hyperparameter of the random forest models was found to be equal to 9, corresponding to setting the number of variables to consider at each tree to 9. We then tested the predictions of the random forest model with this optimal hyperparameter obtained by cross-validation on our trustworthiness and dominance test sets. This test revealed a high performance of the model (trustworthiness: r = 0.85 ± 0.5, t(75) = 14.17, p < 0.001; dominance: r = 0.86 ± 0.05, t(75) = 14.72, p < 0.001; Supplementary Fig. 2; all the reported statistical tests are two-sided).

Validation of the algorithm for modeling trustworthiness and dominance evaluations

To assess the accuracy our trustworthiness and our dominance generator algorithm, we tested their predictions on four different face databases: the Karolinska database (N = 70 distinct faces)[55], the Oslo Face database (N = 185 distinct faces)[56], the Chicago database (N = 520 distinct faces)[57] and the FEI Face database (N = 520 distinct faces)[58]. Given that our model was optimized on avatar faces, comparing our model’s prediction to real participants ratings in a second step allows us to assess whether our model would give overall coherent ratings with those of real human beings. Our first analysis confirmed the significant correlation of the modeled trustworthiness and dominance estimates with the actual participants’ ratings of trustworthiness and dominance ratings on the faces from these databases (except the FEI Face database which did not provide subjective ratings; Supplementary Figure 3). We found significant correlations for both trustworthiness and dominance estimates (trustworthiness: r = 0.22, p < 0.001, dominance: r = 0.16, p < 0.001—N = 768 for each correlation, to not artificially increase the statistical power of this analysis only the neutral and facing version of the faces were used for these correlations), confirming that our model gave trustworthiness and dominance estimates that are coherent with real participants’ evaluations on these traits.

Going one step further, we assessed whether our modeled trustworthiness and dominance were able to reproduce classical findings in social cognition on perceived trustworthiness and dominance, namely: gender effect (females appear as less dominant and more trustworthy than males; trustworthiness: real effect: t(768) = 7.94, p < 0.00; recovered effect: t(972) = 2.67, p = 0.008; dominance: real effect: t(769) = −7.80, p < 0.001; recovered effect: t(972) = −3.63, p < 0.001; Supplementary Fig. 4A, B), emotion effects (angry faces appear as more dominant than neutral faces: t(167) = 9.42, p < 0.001; happy faces appear as more trustworthy than neutral and angry faces: t(167) = 10.64, p < 0.001; Supplementary Fig. 4C, D), head orientation effects (trustworthiness and dominance evaluations for a unique identity are correlated across head orientations: trustworthiness: r = 0.29, t(1500) = 11.51, p < 0.001; dominance: r = 0.34, t(1500) = 13.79, p < 0.001; Supplementary Fig. 4E, F) and age effect (older adults appear as more dominant and less trustworthy than younger adults: trustworthiness: real effect: r = −0.12, t(518) = −2.75, p = 0.006; recovered effect: r = −0.12, t(518) = −2.68, p = 0.008; dominance: real effect: r = 0.32, t(518) = 7.72, p < 0.001; recovered effect: r = 0.16, t(518) = 3.70, p < 0.001; Supplementary Fig. 4G, H)[21,24,59,60].

All these effects were replicated with the modeled trustworthiness and dominance evaluations. In addition, although dominance and trustworthiness were modeled independently, we also replicated the classical correlation between these two traits, further suggesting the importance of investigating trustworthiness conjointly with dominance (effect on participants’ ratings: r = −0.21, t(768) = −5.81, p < 0.001; recovered effect: r = −0.46, t(768) = −14.30, p < 0.001).

Importantly, we further validated our model by replicating the gender effect on all the portraits extracted from a Google image search for ‘women portraits’ vs ‘male portraits’ containing both pictures and paintings (N = 633; trustworthiness: t(632) = 7.89, p < 0.001; dominance: t(632) = −11.79, p < 0.001; Supplementary Fig. 5A, B). We also replicated the gender effect on the official portrait pictures of US representatives (N = 419; gender: trustworthiness: t(417) = 2.20, p = 0.028, dominance: t(417) = −4.74, p < 0.001; Supplementary Fig. 5C, D). Importantly, we also replicated the effect found in the literature that conservative representative appear more dominant than democrat representatives (t(417) = −2.59, p = 0.009; Supplementary Fig. 5E).

Testing the relationship between interpersonal trust and portrait Selfies’ trustworthiness

We tested whether displayed trustworthiness could be used as a proxy for interpersonal trust. To do so, we analyzed the Selfiecity database[29] which includes 3230 selfies posted on Instagram in 2013 from six cities across the world (Bangkok, Berlin, London, Moscow, New York and Sao Paulo; analyzable images: N = 2277[29]).

The identified faces were then individually analyzed by two independent raters who were asked to evaluate, for each picture, the alignment of the OpenFace’s face identification points compared to the real face’s contours (coded as 0 or 1). The sum of these goodness of fit was then used as weights for the analyses. Therefore, only faces for which the two raters agreed that they were not well detected were removed from the analyses. Faces for which the two raters agreed on their good detection had a weight of 2 in the analyses, and those on which they disagreed had a weight of 1.

Importantly, a preliminary analysis confirmed that the trustworthiness computed with our algorithm recovered the gender effect documented in the literature in this image sample too (trustworthiness: t(2275) = 13.80, p < 0.001; dominance: t(2275) = −10.18, p < 0.001; Supplementary Fig. 6A, B). Extracted trustworthiness was analyzed using a linear model taking the sitter’s gender, the city longitude and latitude and the sitter’s dominance as control variables. The effect of two measures of interpersonal trust were used to assess the link between displayed trustworthiness and interpersonal trust, extracted from the European and World Value Surveys[30,31] general social trust question (‘most people can be trusted or you cannot be too careful’; Supplementary Fig. 6C) and the sum of five questions bearing on cooperation (‘how acceptable is claiming government benefits’, ‘avoiding a fare on public transport’, ‘cheating on taxes, keeping money that you have found’, ‘failing to report damage you’ve done accidentally to a parked vehicule’; Supplementary Fig. 6D). As the Selfiecity database is constituted of pictures posted online in 2013, for each country, the most recent vague of the European or World Value Survey was taken (i.e., 2008 for Russia, 2009 for Great Britain, 2011 for the United States, 2013 for Thailand and Germany, and 2014 for Brazil). In line with our hypotheses, people located in places where interpersonal trust and cooperation are higher, displayed higher levels of trustworthiness in their selfies (cooperation level: b = 0.13 ± 0.03, z = 3.67, p < 0.001; trust level: b = 0.81 ± 0.23, z = 3.50, p < 0.001; Supplementary Fig. 6C, D).

Analysis of the National portrait gallery

All the paintings of the National Portrait Gallery were downloaded in high resolution from the NPG.uk website. Information about the sitter’s age at the date of the portrait were also automatically collected. Portraits’ date were automatically coded following the method detailed in the table below (Supplementary Table 2). These values were divided by 100 for the regression analyses such that 1 time unit corresponds 100 years. All the portraits were processed using the OpenFace algorithm. The identified faces were then individually analyzed by three independent raters who were asked to evaluate the model’s goodness of fit based on the points’ position compared to the real face’s contours (coded as 0 or 1). In addition, raters had to note the gender of the sitter. The classification based on the goodness of fit was then used as weights for the analyses. Importantly, in order to ensure that the portraits accurately reflected the level of trust at the time the portrait was painted and to avoid re-interpretation of past historical figures, only portraits painted during the sitter’s lifetime were analyzed (number of analyzed portraits: N = 1962). Portraits’ dates were automatically coded following the nomenclature reported in Supplementary Table 2.

Level of affluence (countries’ GDP per capita) was provided by the Maddison Project[61] and political democratization (Polity 2 index) was provided by the Polity IV project[62]. For the UK, these data exist from 1500 to 2000 for GDP per capita and yearly data from 1800 to 2013 for the democratization index.

In order to keep a maximal temporal resolution, missing values in the GDP per capita and Polity2 indices were completed using the closest previous value, except for the time-lag analyses in which no imputation was made. A total of 1943 data points were included in the analyses looking at the effect of GDP per capita. A total of 1115 data points were included in the analyses looking at the effect of Polity2. Paintings were analyzed using individual linear models (each painting corresponding to one data point), taking the sitter’s gender, age and level of dominance as control variables. Bayes factor analyses were conducted using the BIC approximation, which approximates Bayes factors computed under the unit information prior[63].

Finally, time-lag analyses were conducted to analyze the temporal dynamics between trustworthiness, GDP per capita and democratization. To do so, data were averaged by decades and analyzed at the aggregated level. The model on trustworthiness at decade d included the simultaneous level of dominance at decade d, the linear effect of the time, the delayed levels of trustworthiness and dominance at d-2, and the level of GDP per capita or democratization at d-2. On the other hand, models of GDP per capita or democratization included the linear effect of time, the delayed level of GDP at d-2 and the delayed levels of trustworthiness and of dominance at d-2. For each variable, the model with the delayed variable of interest (GDP per capita or democratization for the trustworthiness models, and trustworthiness for the models on GDP per capita and democratization) were compared with the models in which this variable was removed. Finally, in order to assess the robustness of our effects, we also tested the same models with a delay of one decade instead of two decades (Supplementary Table 3).

Web gallery of art

Data from the Web Gallery of Art (WGA) were analyzed in a similar way as the paintings from the NPG. To better account that the portraits actually reflected the sitter’s willingness to display trustworthiness traits, paintings were geocoded using the painter’s place of activity at the time of the painting. This geo-coding resulted in 19 countries with paintings ranging from 1360 to 1918. As previously, two independent raters categorized the quality of detection of the faces and these evaluations were used as weights in the linear regression (number of analyzed portraits: N = 4106). As for the National Portrait Gallery, the missing levels of affluence and democratization were completed using the previous complete value. The same models as previously were used except that a random effect was included to take the localization of the paintings into account. This resulted, for the analysis of the effect of GDP per capita and democratization in two-level mixed models, taking each painting as an individual data point clustered by the country of production. Correspondingly, for time-lag analyses, we use two-level mixed models but with data aggregated by decades.