Quantifying changes in societal optimism from online sentiment.

Individuals can hold contrasting views about distinct times: for example, dread over tomorrow's appointment and excitement about next summer's vacation. Yet, psychological measures of optimism often assess only one time point or ask participants to generalize about their future. Here, we address these limitations by developing the optimism curve, a measure of societal optimism that compares positivity toward different future times that was inspired by the Treasury bond yield curve. By performing sentiment analysis on over 3.5 million tweets that reference 23 future time points (2 days to 30 years), we measured how positivity differs across short-, medium-, and longer-term future references. We found a consistent negative association between positivity and the distance into the future referenced: From August 2017 to February 2020, the long-term future was discussed less positively than the short-term future. During the COVID-19 pandemic, this relationship inverted, indicating declining near-future- but stable distant-future-optimism. Our results demonstrate that individuals hold differentiated attitudes toward the near and distant future that shift in aggregate over time in response to external events. The optimism curve uniquely captures these shifting attitudes and may serve as a useful tool that can expand existing psychometric measures of optimism.

Introduction

Optimism is a cognitive construct that reflects individual differences in how favorable people expect their future outcomes to be (Carver et al., 2010). Psychological instruments that measure optimism often ask respondents to generalize their expectations over unspecified time horizons (e.g., “I’m always optimistic about my future”) and self-report their attitudes (Dember & Brooks, 1989; Scheier et al., 1994; Schweizer & Schneider, 1997). Optimism measures are generally stable for a given individual at different measurement points (Carver et al., 2010; Matthews et al., 2004), though they have been shown to vary with changes in life circumstances (Chopik et al., 2015; Segerstrom, 2007). Optimism has also been measured at the group level (Mattis et al., 2003) to provide information about the well-being of society (Seaford, 2011). Again, this is typically done via self-report surveys about the generalized future and sometimes about single specific points in time (e.g., 50 years from now) (Gramlich, 2019; Pew Research Center, 1997).

However, individuals can hold different attitudes toward different events in the future: One may be looking forward to a vacation next month, while dreading a dentist visit the week after. Such finer-grained time-indexed trajectories of optimism may tell us more about an individual’s or group’s future-oriented attitudes and behaviors. However, asking people to self-report how their optimism differs towards next week, next month, and next year may not produce behaviorally accurate or stable results due to common validity and reliability issues of self-report scales (Dunning et al., 2004; Epley & Dunning, 2006). Here, we aim to measure and understand the optimism trajectory of large populations by addressing these challenges through the use of big data from social media that captures multiple time-horizons without self-report.

Previous work has used social media data to measure future orientation, a related metric. One group leveraged search engine logs to assess the degree to which societies seek information about the future compared to the past, creating a “future orientation index” which was positively associated with gross domestic product (GDP) (Preis et al., 2012). These results were replicated and expanded with time horizons that differentiate countries by how far into the future people consider; again, higher GDP countries showed a greater future focus (Noguchi et al., 2014). Future orientation has also been measured at a more local level in the United States, where counties with more future-referencing searches have a lower prevalence of HIV (Ireland et al., 2015). These studies use search volume to create their measures, and do not assess the sentiment of the content. Other work has looked at an individual level. Classifiers have been developed that score social media users on future orientation, and these scores correlate positively with conscientiousness, satisfaction with life outcomes, and income (Park et al., 2017; Hasanuzzaman et al., 2017), similar results to those obtained with psychometric scales related to future orientation (Strathman et al., 1994; Zimbardo & Boyd, 2015). Other groups have used machine learning to classify Twitter users as well as individual tweets as optimistic or pessimistic, consistently finding robust accuracy (Alshahrani et al., 2020; Ruan et al., 2016; Caragea et al., 2018). These studies of optimism on Twitter, like those completed with individual difference measures, operationalize optimism as outlook toward the generalized future and do not differentiate between time points as we do here.

In constructing this new measure, we take inspiration from an economic indicator that reflects investors’ optimism regarding different future time horizons: the Treasury bond yield curve. Treasury bonds are fixed-income securities backed by the United States government (Campbell, 1995) that mature at different times and will often have different rates of return, or yields, based on the date of maturity. Treasury bond yields are adjusted based on supply and demand manifested through their sale at auction to investors, which capture investors’ levels of certainty or uncertainty about the health of the economy at different times in the future (the different maturity dates). With greater uncertainty (akin to lower optimism), investors demand greater returns for their investments. This can be summarized with a plot that relates the current specified yield on a particular bond to its time (e.g., in days) until maturity on a given date (U.S. Department of the Treasury, 2020). This plot, the yield curve, is usually upward sloping, with bonds that mature in the more distant future offering higher yield rates than those that mature sooner (Fig. 1, typical curve). The upward slope reflects that investors have more positive expectations for short-term investments and greater uncertainty toward the more distant time points. During some periods, this curve inverts, and there are higher yields for bonds that mature sooner compared to later (Fig. 1, inverted curve, see Supplemental Fig. S1 for another example). This inverted yield curve reflects a widespread shift in economic uncertainty, and its occurrence often precedes economic recession (Stock & Watson, 1989). In fact, an inversion of the yield curve has preceded every US recession in the past 50 years by about 6 to 16 months. The level of the curve as a whole, regardless of its slope, can vary as both short- and longer-term yields increase or decrease.

Fig. 1

Treasury bond yield curves for a typical and inverted period, showing all yield rates (log scaled by days to maturity) and best fit lines

To assess population-level optimism regarding multiple time horizons, we analyze the sentiment of tweets that reference different points in the future, as a parallel to the different dates to maturity of bonds. While the Treasury bond yield curve conveys information about investor perceptions of future risk and uncertainty, the “optimism curve” we construct here depicts how positively (or negatively) people discuss the future on social media, creating a measure of collective future optimism that is temporally fine-grained without relying on self-reports.

We use the optimism curve to study the relationship between the sentiment of language used and the time point referenced, and whether this relationship itself changes over time with major events as seen for the bond yield curve. Specifically, COVID-19, declared a pandemic on March 11, 2020 (World Health Organization, 2020), has caused massive disruptions to economies and social structures worldwide (Petrella et al., 2020, March 20; Hughes & Andrews, 2020, March 27; Liu et al., 2020; Dong et al., 2020; Valdez et al., 2020). The pandemic has had major impacts on individuals’ mood and mental health that we look for as changes in the optimism curve, reflecting societal shifts in positivity toward the near and distant future compared to typical levels in previous years.

Materials and methods

Data and materials availability

All data and source code used in this study are available in deidentified form in a dedicated and open GitHub repository: https://github.com/CalvinIsch/optimism-curve. Any additional information with respect to the data used in this study will be made available from the corresponding author upon reasonable request, provided this information can be made available in deidentified form.

Samples and limitations

Our analysis is based on a collection of tweets containing future-referencing phrases from August 1, 2017 to November 1, 2020 (see daily count time series in Supplemental Fig. S2). We collected the data using the open analytics platform IUNI Observatory on Social Media (OSOME) (Davis et al., 2016) – a tool that captures a random 10% sample of all public tweets. Location filters were not used when collecting the sample. We searched for tweets referring to 23 different time points with one of five commonly used phrases that unequivocally express the specific point in time relative to the date that the tweet was posted (e.g., “in 3 days”, “2 weeks from now”, see Supplemental Table S2 for a complete list of phrases and Supplemental Fig. S3 for other time points considered), with time points spread roughly equally over log-scaled distance into the future. This means we collected more near-future tweets, and progressively fewer distant-future ones. We collected high-frequency phrases rather than an exhaustive list of all grammatically valid future references. We also explored whether tweets referencing different time phrases that indicate the same time point (i.e., “12 months” and “1 year” from now) showed similar sentiment across time, finding that they do (see Supplemental method 1). With these criteria, we found a total of 3,568,985 tweets after excluding retweets. Tweets that did not have scorable sentiment (see next section, n = 1,267,813) were removed. We ran the remaining tweets through the bot-detection service Botometer (Yang et al., 2020) and removed all tweets from users with a rating greater than .5 which indicates that these accounts are likely bots (Supplemental Fig. S4 contains the proportion of bot tweets for each future reference). We retain 2,220,462 tweets after these steps. Table 1 contains the number of tweets with every phrase after these exclusions.

Table 1

Number of tweets with each time point in sample, split into near-, medium-, and far-future groups. For each time point, we searched adjacent words to select only tweets that explicitly reference the future, e.g., “2 days until”, “2 days from”, “in 2 days”, and then grouped tweets by time point

Near	Medium	Far
2 days (N = 230,441)	6 weeks (N = 22,447)	12 months (N = 16,705)
3 days (N = 195,239)	2 months (N = 86,198)	2 years (N = 109,415)
4 days (N = 110,529)	3 months (N = 85,809)	3 years (N = 97,571)
5 days (N = 105,692)	4 months (N = 40,575)	4 years (N = 78,585)
7 days (N = 74,077)	5 months (N = 29,382)	5 years (N = 123,712)
10 days (N = 99,878)	6 months (N = 87,406)	10 years (N = 132,512)
2 weeks (N = 236,062)		20 years (N = 84,215)
3 weeks (N = 99,853)		30 years (N = 46,387)
4 weeks (N = 27,772)

Our sample and selection process come with limitations. We chose a high-precision criterion, namely that the tweets match a specific future-referencing statement, at the cost of capturing fewer of all relevant statements. Our Twitter sample is furthermore not a representative sample, and can as such be subject to platform-specific bias. However, we are not making claims with respect to the absolute levels of sentiment regarding particular future times, but the comparative slope of the optimism curve for the same sample of Twitter data. To enable drawing conclusions about absolute levels of optimism, future research should cross-validate these results with other social media platforms and with vetted, representative samples.

Statistical analysis

To assign each individual tweet a numerical sentiment score, we used an open-source, accurate Twitter sentiment analysis algorithm: the Valence Aware Dictionary and sEntiment Reasoner (VADER) (Gilbert & Hutto, 2014). VADER contains an empirically validated lexicon of 7516 of the most frequently used English words and symbols (including jargon, abbreviations, and colloquial language). These terms were rated during VADER calibration by multiple independent human coders with respect to their valence (polarity and intensity) in the context of micro-blog content (e.g., Twitter). VADER also employs heuristics to recognize negations, hedging, boosters (e.g., “very”), and style common to Twitter. Because VADER uses heuristics and a simple lexicon it is self-contained and domain agnostic, performing well on novel datasets. A recent analysis comparing 22 sentiment analysis packages found that VADER produced the highest accuracy on a dedicated Twitter dataset (F1pos = 99.25, F1neg = 98.33, using the F1 measure of classification effectiveness), outperforming commonly used tools such as SentiStrength and Linguistic Inquiry and Word Count as well as individual human raters (Ribeiro et al., 2016). With these advantages, VADER has been used in several contexts, e.g., to study emotions (Fan et al., 2019; Bathina et al., 2021), to evaluate patient experience with health care across the United States (Sewalk et al., 2018), and to assess public attitudes during the COVID-19 pandemic (Valdez et al., 2020).

In this study, we analyze the compound VADER score, a unidimensional measure of a tweet’s valence between -1 (very negative) and 1 (very positive). A few examples can illustrate the ability of VADER to gauge a tweet’s sentiment: The compound sentiment score for the sentence “This ice cream is good” is 0.4404, whereas for another sentence (“This ice cream is very good”) the score is 0.4927. A negated version of this sentence (“This ice cream is not good”) has negative valence (-0.3412), and for another more matter-of-fact statement (“There is ice cream on the table today”) the score is 0. Like most sentiment analysis tools, when a tweet contains no words from the extensive VADER lexicon, VADER returns a sentiment rating of exactly zero. This does not imply the tweet has no sentiment, but simply that it did not match any of VADER’s lexicon words. To avoid mistaking tweets that have a VADER rating of exactly zero because no lexicon words matched (> 90% of tweets with VADER score of 0 in our dataset) with tweets that actually do have neutral (0.0) sentiment, we removed all zero-rated tweets from our analysis (N = 1,267,813, 36% of our sample, see histogram of remaining VADER scores in Supplemental Fig. S5). However, all reported results remain significant if these tweets are included (see Supplemental method 2).

We first binned tweets by the month they originated in, and then for each time point (e.g., “3 weeks” in the future) we calculated the mean VADER score of tweets that reference that time point for every monthly bin. We regress these monthly means on the distance until the time point mentioned (in days, log scaled) to create the optimism curve for each month worth of tweets (see Supplemental Fig. S6 for a visualization of this process). We plot the slope of these monthly best-fit regression equations along with the monthly mean sentiment of all future references to explore how these measures change over time. Finally, we also calculate Pearson correlations between VADER score and time point referenced before and during the first month of the pandemic (See Supplemental method 3 for alternative approach based on relative sentiment).

To compare optimism toward the near, medium, and far future, we also split the tweets into three groups: near-future (references to four weeks or fewer from now), medium-future (references greater than 4 weeks and less than 12 months from now), and far-future (references 12 months to 30 years from now). As with the optimism curve, we calculated the monthly mean VADER score by each phrase, and these monthly means became the observations included in each group (see Supplemental Table S3 for distribution statistics). To compare these groups, we ran one-way ANOVAs on the measures both before and during the pandemic. We also used model segmentation, splitting the time points into four groups based on the best-fit quartic model, to explore these non-linear trends (see Supplemental method 4), finding similar results.

Results

Time-indexed measure of optimism shows differentiated assessments of near and distant future

We first analyzed the 1,624,426 tweets that met our criteria and originated before March 2020, the month COVID-19 was declared a pandemic. We found mean VADER scores for each time point and then ran hierarchical regressions comparing the goodness of fit for polynomials of degree 1–4 of mean VADER score on log-scaled distance in the future (see Supplemental Table S4). A linear function fit the data well (R2 = 0.55, p < .001), and although a fourth-degree polynomial improves fit slightly, a five-fold cross-validation task reveals that this model overfits our sample relative to the first-degree polynomial we selected (See Supplemental Fig. S4). For this reason, here we rely on linear models. In the section “Medium-future references show lowest sentiment”, we employ alternative approaches (i.e., grouping into three sets of time points) which address the possibility of non-linear trends in the data.

Longitudinal analysis reveals inversion of optimism during the COVID-19 pandemic

To test the stability of the mentioned linear model across the time range under investigation, we repeated the same linear fit for each month of data in our Twitter data range separately. These monthly results each show moderate to strong negative correlations (range r = -.44 to -.82, range p = .04 to p < .001, see Supplemental Fig. S8 for scatter plots and best fit lines for each month), suggesting that individuals consistently talked about the near future with greater positivity than the distant future during the entire 31-month period before COVID-19 was declared a pandemic. In contrast, the single-month correlation becomes slightly positive in the month after the pandemic began (r = .14, March 11 to April 11, 2020), as people register considerably less positivity toward the near and medium future than they had before, while maintaining similar levels of positivity toward the distant future—a reversal reminiscent of that between the typical and inverted Treasury bond yield curves (Fig. 1). This inversion is illustrated in Fig. 2 with the optimism curves for a typical month before (Y = -.033X+ .23) and an inverted month during (Y = .005X + .09) the pandemic. We used correlations to see the strength of the relationship between time point referenced and sentiment. Throughout the remainder of the text, we will look at the slope of the best fit line to see the mapping between these variables.

Fig. 2

Typical and inverted optimism curves. A downward slope indicates lower sentiment towards more distant future dates and was typical before the COVID-19 pandemic. In March 2020, the curve inverted and became upward sloping, indicating lower sentiment toward the near compared to more distant future. Means, best fit lines, and error bars representing 95% confidence intervals are shown

We next plot the slope of the optimism curve for every month of available data, along with the mean monthly VADER score for all of our future-referencing tweets (Fig. 3). Together, these two measures depict changes in general optimism and how that optimism is distributed toward the near and far future. While average sentiment initially drops and remains atypically low during every month of the pandemic, the slope of the optimism curve initially inverts and then returns to normal levels after the first few months.

Fig. 3

The brown time series displays the monthly optimism curve slope and the blue time series displays the mean VADER score for each month with available data. Time series have a gray background during the COVID-19 pandemic. The four-quadrant plot to the right contains both measures for each month with the inverted months colored red

In March 2020, when COVID-19 was declared a global pandemic, the mean VADER score fell to 0.11, 6.42 standard deviations below the mean monthly value. This indicates that during this month users on Twitter talked about the future overall with much less positivity than normal. In April, the optimism curve reached a slope of 0.002 (4.51 standard deviations above the mean). Together, these changes suggest that the mean decrease in positivity is primarily caused by a drop of sentiment for tweets discussing the near- and medium-term future (as seen in Fig. 2). The near-future references begin to bounce back from this initial drop after these first months, but the medium-future references remain relatively low (see Fig. 4).

Fig. 4

Time series with mean VADER score for tweets containing near-future (blue line, four weeks or fewer from now), medium-future (green line, more than four weeks and less than 12 months from now), and far-future (orange line, 12 months to 30 years from now) references using a moving average of 7 days. March 11, the date COVID-19 declared pandemic, is marked by the vertical gray line

Medium-future references show lowest sentiment

While we use a linear model for these analyses, Figs. 2 and 4 show that, at the beginning of the COVID-19 pandemic, tweets referencing the medium-term future indicate lower overall sentiment compared to those that reference either the near- or far-term future, and this pattern has continued since (see Supplemental Fig. S9). To quantify this dip, we compare the monthly mean sentiments of near-, medium-, and far-future references with a one-way ANOVA both before and during the pandemic (see Supplemental Table S3 for distribution statistics). Before the pandemic, these groups of time points differed significantly (F = 293.2, p < .0001, n = 713, df = 2). Near-future references had the most positive VADER score (M = .22) and were significantly higher than medium-future references (M = .15, F = 359.0, p < .0001, df = 1) and far-future references (M = .14, F = 535.4, p < .0001, df = 1). During the pandemic (March - October 2020) the groups again differed significantly (F = 81.1, p < .0001, n = 184, df = 2). The near-future tweets are still the most positive (M = .17) and are significantly higher than both the medium-future (M = .07, F = 124.9, p < .0001, df = 1) and the far-future (M = .13, F = 34.3, p < .0001, df = 1). During this interval, the drop is clearly seen for the medium-future tweets, which are also significantly less positive than the far-future tweets (F = 73.4, p < .0001, df = 1). For a more detailed perspective into the future references that drive this result, see Supplemental Fig. S10.

Discussion

In this paper, we proposed an “optimism curve” that compares levels of optimism toward multiple time points in the near, medium, and far future. Similar to typical Treasury bond yield curves, where investors demand higher yields for distantly maturing bonds, tweets referencing the distant future generally carry less positive sentiment than those that reference the near future. We speculate that this may be related to the propensity of individuals to look at the near future with greater concreteness and certainty than the far future (Liberman et al., 2007; Trope & Liberman, 2010). They may thus tend to look forward to more specific positive outcomes in temporally closer time points.

While the patterns described here pertain to changes in the slope of the optimism curve over time, it is not yet clear whether this curve is better described as linear or a more complex function, similar to those between other psychological measures and distance in time (Snefjella & Kuperman, 2015). In this sample, during the pandemic, tweets that reference the medium future had lower sentiment than those that referenced the near or far future. This lasting dip in medium-term sentiment decreased the linearity of the optimism curve during this period, and it may capture the interaction of several psychological variables including fear, uncertainty, and concreteness. Future research will be directed at identifying the possible drivers of the optimism curve shape.

Our analysis shows that the typical pattern of decreasing optimism toward the more distant future can temporarily invert. The Treasury bond yield curve inverted in February and March of 2020 (previously in 2007 and 2019), which could be interpreted as a signal that investors at that time were (correctly) anticipating an economic contraction during the COVID-19 pandemic. Likewise, at that point in time we also observed an inversion of the optimism curve, reflecting that Twitter users collectively dropped in optimism toward the near and medium future relative to the far future. Inversions of the bond yield curve are generally thought to be a leading indicator of future economic contractions, since they indicate that investors are collectively assuming that risk levels will be higher in the near future than the more distant future. Whether this is also the case for the optimism curve will need to be addressed in further research. Possible applications of the optimism curve as an indicator of social and economic changes are enhanced by the ability to monitor it continuously, as it is derived from large-scale social media data updated in real time.

This paper uses Twitter data to construct the optimism curve. Such a data set is advantageous because it measures implicit statements of positivity toward specific future time points, avoiding many of the pitfalls associated with explicit self-report measures. However, these data are limited in that they come from a sample–Twitter users–that is not representative of all individuals in any particular location. Additionally, because we use an English sentiment analysis package, our sample only captures the Anglophone Twittersphere. Future work should assess both the convergent validity of the observed optimism curve, comparing it with other methods of measuring optimism, and its external validity, exploring whether the same curve emerges in other contexts including diverse populations and in languages other than English. How well do self-report scales of optimism match the valence of speech referencing specific future time points as measured here with social media data? How similar are various populations’ optimism curves? Do differences in these curves predict differences in behavior at an individual or collective level? These questions provide exciting ongoing directions for this line of research.

A decline in positive emotional valence in near- and medium-future-referencing tweets may indicate upcoming negative psycho-social outcomes. A recent study found that, at the individual level, emotional valence of future goal-directed imaginations predicted levels of well-being two months later (Gamble et al., 2021). Our analysis provides similar results at a societal scale: The drop in near and medium future positivity at the beginning of the pandemic corresponds with increased mental health issues that lasted for months into the pandemic (Khan et al., 2020; Daly et al., 2020). As such, the optimism curve may be viewed as an indicator of the state of current and upcoming mental health across society. Further research should assess the extent to which the optimism curve can point to future changes in decisions in social, financial, and other realms.

The large proportion of people who are active on social media (Pew Research Center, 2019) also means this optimism curve approach can capture future sentiment well beyond that of bond investors alone. This sample could further be stratified to focus on disposition towards the future for specific demographics or locations. The shape, slope, spread, and change of the optimism curve could thus provide a rapid and nuanced measure of societal optimism to help us understand how the events of today impact the thinking about tomorrow and beyond for millions of individuals.

Below is the link to the electronic supplementary material.

(PDF 3.46 MB)