The effects of social feedback on private opinions. Empirical evidence from the laboratory.

The question of how people change their opinions through social interactions has been on the agenda of social scientific research for many decades. Now that the Internet has led to an ever greater interconnectedness and new forms of exchange that seem to go hand in hand with increasing political polarization, it is once again gaining in relevance. Most recently, the field of opinion dynamics has been complemented by social feedback theory, which explains opinion polarization phenomena by means of a reinforcement learning mechanism. According to the assumptions, individuals not only evaluate the opinion alternatives available to them based on the social feedback received as a result of expressing an opinion within a certain social environment. Rather, they also internalize the expected and thus rewarded opinion to the point where it becomes their actual private opinion. In order to put the implications of social feedback theory to a test, we conducted a randomized controlled laboratory experiment. The study combined preceding and follow-up opinion measurements via online surveys with a laboratory treatment. Social feedback was found to have longer-term effects on private opinions, even when received in an anonymous and sanction free setting. Interestingly and contrary to our expectations, however, it was the mixture of supportive and rejective social feedback that resulted in the strongest influence. In addition, we observed a high degree of opinion volatility, highlighting the need for further research to help identify additional internal and external factors that might influence whether and how social feedback affects private opinions.

The same epoch that has witnessed the unprecedented technical extension of communication has also brought into existence the deliberate manipulation of opinion and the “engineering of consent”. There are many good reasons why, as citizens and as scientists, we should be concerned with studying the ways in which human beings form their opinions and the role that social conditions play.

([1], p. 31)

Introduction

Solomon Asch had good reason to encourage fellow social scientists to join him in his attempt to understand why and how human beings develop and change their opinions. His famous group experiments on conformity revealed how willing participants were to conform to group behavior and abandon their own views no matter how objectively skewed the majority opinion was [1, 2]. Confronted with the simple task of visually comparing the length of lines, a remarkable number of participants adjusted their estimates in the direction of those given by the other subjects in the room, who had secretly been instructed to answer erroneously. The results, highlighting the potential impact of social influence on opinions, led to a question which has inspired many and will be guiding the paper at hand: “Exactly what is the effect of the opinions of others on our own?” ([1], p. 31).

The extent to which Asch’s conformity experiments answer this question is, however, limited. This becomes apparent by the reasoning that yielding participants offered when asked about their change of heart. Only very few fully adapted and simultaneously claimed not to have been aware of the influence of the majority opinion on their own. Most of the yielding subjects admitted having deliberately adopted the unanimous estimate of their group members. The subsequently reported reasons for their individual decisions varied greatly: While some were honestly convinced that their own perception was flawed, others confessed that they conformed due to fear of judgment.

Evidence from another series of classic experiments on social influence reveals that Asch’s participants had good reason to believe that non-adjustment might expose them to undesired outcomes. When observing influencing behavior in groups, George Homans discovered that participants insisting on minority positions were more likely to face social exclusion or punishment from those they disagreed with [3]. These findings corroborated his postulate that individuals find value in other people agreeing with them and fit well in his theoretical frame-work of social exchange which provides a pathway to understanding social influence on opinion change. Central to the argument of social exchange theory is the assumption that individuals, when interacting with each other, award the activities of their fellows with either reward or punishment. Rewards, or reinforcers as Homans calls them, make a certain behavior more likely, while the withdrawal of reinforcers or the punishment of behavior leads to a decline in likelihood. In situations of social exchange, intangible goods such as sentiments can serve as either reinforcers or punishers. Among these, social approval is of particular interest for it is possible to encourage a broad variety of human activities by rewarding them with this kind of affirmation. In combining these thoughts, Homans concludes that people give social approval to others that have given them an activity they value. This reinforcement in turn makes it more likely that the others will repeat the action in question. As similarity in behavior is one feature that individuals find valuable, open disagreement thus can be expected to result in punishment to the point of exclusion.

The assumption that individuals adjust their public opinion expression due to fear of social isolation is also central for the spiral of silence theory put forth by Elisabeth Noelle-Neumann [4]. According to her conceptualization, people are in a constant and mostly subconscious state of monitoring opinion sources. When deciding on whether to speak out or to remain silent within a certain social environment they let their subjective impression of public majority opinion guide them in figuring out which opinions are prevalent and therefore safe to express. According to Noelle-Neumann, learning from knowledgeable others can be identified as an important cause for changes in publicly expressed opinions.

Nowadays, this idea seems worthy to reconsider as the vast spread of Internet usage has enabled people to observe and exchange information as well as opinions at an unprecedented rate; a development which has raised hopes and fears alike among scholars of opinion dynamics. The optimists on the one side expect the emergence of a new online public sphere in which citizens will experience exposure to political discussion and cross-cutting information, which has been known for it’s moderating effect on political dissent and stimulating impact on political participation [5-8]. The skeptics on the other side fear that individuals will either refrain from exchange with those that think differently or embrace more and more opposite standpoints when interacting with them [9, 10]. They expect online communication to lead to an increase in opinion polarization, a state that is feared by many for its undermining consequences on social and political stability [11].

Public expression and private opinion

The approaches and findings outlined above share a more or less explicit focus on publicly expressed opinions. But social scientists have long been aware of the fact that “the opinions a person expresses publicly may diverge in varying degrees from those which he holds in private” ([12], p. 427). This raises the question of what we can actually learn about the effects of social influence from simply observing adjustment of a publicly expressed opinion which might be motivated by group pressure.

It was only a few years after Asch that Kelman called for a better understanding of the conditions under which opinions are formed and changed, adopted and abandoned, and likely to be expressed [13]. His own theoretical reflections started out from a rather basic but useful distinction between two potential outcomes of social influence processes: public conformity and private acceptance. While the former is characterized through a verbal and situational adaptation that is not necessarily preceded by a conversion of the individual’s beliefs, the latter represents a more general and enduring internal change of opinion. Kelman expatiated three processes by which people adopt opinions in the wake of social influence: compliance, identification, and internalization. The latter, in particular, is associated with private acceptance, as it occurs when a person finds the content of an influence intrinsically rewarding and “inherently conductive to the maximization of his values” ([13], p. 65). Publicly expressed opinions, however, that are due to compliance with surrounding others are prone to vary with the situational circumstances [14]. The revealing finding of how, as in Asch’s experiments [1, 2], vastly different motivations on the individual level can lead to the same type of observed behavioral outcome exemplifies the extent to which the factual effects of social influence can go by unacknowledged when simply measuring differences in public expression. Capturing the changes that occur within the individual is thus essential for making predictions of subsequent behavior, or when trying to understand whether and under what conditions opinions form, persist, change, and translate into action. Therefore, when concerned with long-term changes in internal convictions and subsequent behavior, researchers have to turn their attention towards analyzing the changes in private opinions, the ones that individuals actually hold.

Reinforcement learning from social feedback

The paper at hand aims to connect these well known challenges of opinion research with the new ones that online communication has brought about. In doing so, we find findings familiar from opinion dynamics research also in the emerging patterns of online communication. In the former, researchers are struggling to explain the persistence of opinion differences between interacting individuals which is at odds with both, empirical evidence from social psychology that shows that people tend to assimilate their opinions to those of people they interact with as well as classical social influence models that predict opinion convergence among interacting individuals in the long run [15-17].

In order to explain these phenomena of opinion divergence, a number of opinion exchange models and social influence mechanisms have been suggested. One of the most recent attempts was the introduction of a reinforcement learning mechanism that is based on social feedback [18, 19]. The affective experience-based approach centers around Homans’ idea that individuals repeat behavior that is rewarded by others [3] and argues along the same lines as Noelle-Neumann [4]: When interacting and expressing opinions within a given social environment, individuals receive social feedback in return. Since human beings generally are sensitive to approval and disapproval [3], they subsequently use the judgments they receive to develop an idea about the prevalent opinions in their social neighborhood and internally evaluate available opinion alternatives. The positive experience of supportive social feedback is intrinsically rewarding and thus leads to a strengthening in the internal attachment to the expressed opinion. Negative, i.e. rejective social feedback, correspondingly, results in decreased adherence. Within the course of such a reward-driven reinforcement process, individuals not only reevaluate the opinions expressed and learn which are safe to express in their neighborhood. They also internalize the expected opinion and integrate it with their existing values until it “gradually becomes independent of the external source”, as Kelman ([13], p. 66) put it, as well as independent of the possibility of observation. Reinforcement learning from social feedback is therefore expected to go beyond effects of mere public conformity. It rather leads to actual changes in internal convictions and, ultimately, in private opinions.

Research questions and hypotheses

Yet up to now, there is no empirical validation for the presumed effects of social feedback. Echoing the ideas and questions that have driven research on social influence and opinion dynamics alike, our empirical approach is motivated by the following research questions:

Does social feedback yield any relevant effect on private opinions? And furthermore, does any potentially resulting change of opinion correspond to whether the social feedback was supportive or rejective in its nature?

From the theoretical groundwork that was presented in the previous section, and in order to provide first empirically supported answers for our research questions, we derived and experimentally tested the following basic hypotheses.

H1 If social feedback is perceived as positive, the private opinion is strengthened and shifts further in the direction of the original position.

H2 If social feedback is perceived as negative, the private opinion is weakened and shifts in opposite direction of the original position.

Suppose a private opinion is represented by a person’s position on a symmetrical scale ranging from complete disagreement to complete agreement on an issue, with the neutral position in the middle. In that case our hypotheses imply the following empirical consequences: If H1 is correct and a person holds (A) a disagreeing private opinion of a certain extent, receiving positive, i.e. supportive social feedback leads to more disagreement. Analogously, if the same person holds (B) an agreeing private opinion, she would agree even more following the positive feedback. In contrast, if H2 is correct a person with (A) a disagreeing private opinion responds to negative, i.e. rejective social feedback by moving towards the anticipated position of the person giving the feedback and thus taking a more agreeing position than before. In the case of an originally (B) agreeing private opinion, on the other hand, it is expected that the extent of agreement will be reduced.

H3 If social feedback occurs in a balanced mixture, the effects of positive and negative judgments cancel each other out and the private opinion will not be affected.

Since it is assumed that neither positive nor negative judgments are superior to their respective counterpart type of social feedback in terms of influential strength, we additionally propose a third hypothesis H3, according to which an equal amount of positive and negative judgments does not result in a change of opinion. Consequently, individuals who receive such mixed feedback from their social environment are expected to act just as they would have in the counterfactual state, in which they had received no social feedback at all.

Experimental study

Design and sample

In order to test these hypotheses we conducted a randomized controlled laboratory experiment. The study consisted of three parts which were carried out at three consecutive points in time. While participants’ initial (pretest) and final (posttest) private opinions were measured through anonymous standardized online surveys at times t1 and t3, the core part of the study, which was the attempted manipulation of a particular private opinion through exposure to social feedback, took place in our laboratory at t2 (see Fig 1).

Fig 1

The course of the study.

Randomization was realized in two steps, as we first randomly assigned participants to either the control or the treatment group and in a second step randomly assigned treatment group participants to one of three different experimental conditions. Subjects had registered as voluntary test persons for experimental research at the Leipzig Experimental Laboratory for Social Sciences (LEx). Within the first e-mail, subjects were invited to participate in a multi-part study, which they could start off by completing the linked online survey. A 20 Euro compensation fee was granted to volunteers who would participate in at least two parts of the study. Wording left the overall number and intention of the study parts open, so that members of neither control nor treatment group knew about the existence of the other group or a variation in procedure. The actual treatment process started after the first online survey was completed, with treatment group members taking part in one of the 18 laboratory sessions. The second online survey concluded the experimental process. A total of 270 people participated in all parts of the study to which they were invited. Overall, the study spanned over a period of 23 days. Since time spans between participation in the lab and the posttest could range from zero to 14 days, the study design allows the observance of durable and integrated treatment effects on actual private opinions which go beyond mere situational and spontaneous shifts in publicly given responses.

To further ensure this crucial study feature, the explicit assurance of permanent anonymity played a prominent role throughout the whole study process. Within the online surveys it increased the likelihood of undistorted and honest responses by the participants, and, therefore, ensured the assessment of what we refer to as private opinions. In the laboratory, the combination of an anonymous setting with a computer-assisted treatment process, which allowed to control all of the content to which subjects were exposed to, made sure that potential social feedback effects would not be obscured by characteristics of other subjects present. This was of particular importance, as it has long been established that socially significant characteristics affect the ability of individuals to assert themselves and convince others in groups [20-24]. In social influence processes individuals thus may show compliance or adaptation because they regard the influencer as a credible source of information or because they want to meet anticipated expectations [13]. To arrive at respective assessments, however, individuals must have access to relevant characteristics of the influencing group or person. In fact, removing social cues like race, gender, age, organizational position, etc. has been shown to cause an increased equality of influence across status and expertise in groups [25]. Consequently, by excluding this type of influential variables the presented experimental design ensured high internal validity while also resembling the rather anonymous setting of online communication. It is, therefore, expected to allow for an investigation of the isolated effects of social feedback on private opinions.

Measurement of private opinions

Measuring the impact of social feedback on private opinions required a study topic that was neither subject to general social consensus nor strongly polarized. It also had to be common as well as emotionally charged enough for subjects to care about other participants’ judgments. Taking these considerations into account, we chose to construct the experiment around an alleged study on “the relationship between humans and animals”; a topic that has been subject to a widespread, morally loaded, and rather emotional public debate in recent years.

Private opinions were measured using the Speciesism Scale, an instrument that intends to capture humans’ discriminatory inclinations towards members of other species [26]. The scale was at the core of both online questionnaires at t1 and t3, which were realized by means of SoSci Survey [27] and were completed at a time as well as in an environment of the individuals’ choice. For each of the 23 items of the Speciesism Scale participants were asked to indicate the extent of their disagreement or agreement on a slider ranging from 0 (complete disagreement) to 100 (complete agreement). Since we were aiming to use an outcome variable that was as evenly distributed as possible at the outset of the study, the opinion distributions of each item, as measured at t1, served as a basis for choosing the target item, i.e. the private opinion that participants would receive social feedback for after expressing their opinion during a laboratory session. We decided to use the following target item:

“The killing and eating of animals is part of human nature.”

Laboratory treatment process

The laboratory sessions were conducted in our computer pool at time t2. Participants were seated at individual work stations, which were isolated by three-sided walls. On average, 12 participants were present in each lab session. They were told that the computer would divide all attending participants randomly into groups of three, who would then interact in three consecutive rounds of opinion exchange. It was stressed that the identity of group members would never be disclosed. The opinion exchange process, as it was presented to the participants, would start with one of the members of each group giving an opinion on a particular statement which would in the following be presented to and judged by the two other group members. Their judgments would then be presented to the opinion giving person only. Reportedly, the entire process was to be repeated three times until each one of them had stated an opinion and judged the opinions of both other group members once.

However, in reality there was no group formation process and no interaction between participants whatsoever. Instead, and unbeknownst to the attending individuals, each one of them went through the same process in exactly the same order, stating their opinion on the target item at the beginning of the first round and receiving two pre-formulated feedback statements at the end of it. All subjects were asked to indicate their opinion on the target item on a 0 to 100 slider in the same way as they had done in the first online survey. This time, though, they were told that the opinion would be presented to the other two group members afterwards. Whilst waiting for the alleged judgments of those fictional group members, participants had the opportunity to further explain their position on a waiting screen, with the sounds of their keyboards giving the impression that social feedback from others was being written down at the same time. After five minutes had passed, the anonymous social feedback statements of the alleged other group members were presented automatically. All of those pre-formulated sentences were completely randomly and independently assigned by the computer. This random assignment of social feedback thus has to be considered as the experimental treatment and second randomization step of the presented study. The second and third round of each laboratory session were staged in order to maintain the illusion that the experiment was conducted as presented in the beginning. The laboratory feedback process was created and carried out by means of z-Tree [28].

Overall, 20 different social feedback statements were distributed among which ten were positive (supportive) and ten were negative (rejective); see S1 Table for a complete list. Participants were simultaneously exposed to two judgments whereby upholding the illusion that they were interacting with two other group members, namely “A” and “C”. The feedback statement assignment resulted in three different treatment groups with participants receiving either two positive (positive feedback condition), two negative (negative feedback condition) or a combination of one positive and one negative feedback statement in varying order (mixed feedback condition). While having varying combinations of social feedback statements subsumed under the labels of each treatment condition could be argued to have introduced treatment heterogeneity, this procedure was necessary as it reduced the risk of participants seeing through the deception afterwards. At the end of the second online survey, participants were asked to evaluate to what degree they perceived each of the 20 statements used in the lab as positive or negative. We found that only in four single cases a participant’s perception of a certain statement differed from our assessments.

Ethics statement

The basic terms and conditions of participation were presented during the LEx online registration process and accepted by subjects upon enrollment. In order to register, people had to be at least 18 years old and agreed that the evidence that was gathered during experiments would be used for scientific research. They are informed that participation is totally voluntary and that there is a financial compensation which varies between studies. At the LEx, participants are free to decide whether to accept an invitation and participate in a particular study. When participants begin taking part in a study including laboratory sessions, they are mostly unaware of the type of process that awaits them in the lab. We have carefully considered the ethics of conducting the study as described above: In order to be able to pursue the research as intended, it was necessary to achieve a situation in which the participants did not know about the existence of control and treatment groups as well as the actual processes in the laboratory. However, after completing the last part of the study, all members of the treatment groups had been informed about the actual study process and given the opportunity to withdraw their permission for data usage. Participants were also able to withdraw from the study at all times. Contact information of contact persons was made available at each step. The design of the data collection processes, both during the online surveys and in the laboratory, ensured that the information provided could not be assigned to the participating persons at any time. In order to ensure responsible handling of study subjects, a study draft was submitted to the Ethical Committee at Medical Faculty of Leipzig University. The Ethics Committee reviewed the experimental design under ethical, medical-scientific and legal aspects and confirmed that the design as well as the mode of operation comply with the legal regulations and relevant ICH-GCP recommendations for risk-benefit assessment of scientific investigations in humans. The full statement of the Ethics Committee was provided upon submission.

Results

Descriptive and inferential statistics

In order to control for potential random confounding that might have occurred despite randomization, we measured various sensible covariates such as age, student status, sex and diet. As we intended to take potential distorting effects of participant personality into account, we furthermore included the Big Five Inventory Short Scale (BFI-10 [29]). It is established that personality traits as measured by the Big Five Inventory have a great influence on attitudes and behavior [30, 31]. With respect to the nature of our study, we wanted to make sure that personal characteristics do not cause any unobserved systematic differences on the effect of social feedback on private opinions. This approach seems to be warranted as research has already shown that Big Five personality traits have an influence on feedback seeking behavior [32] and that characteristics such as extraversion and agreeableness have an influence on affective polarization towards an opposing opinion group [33]. An overview of all variables can be found in S1 Appendix. The sample analyzed includes all subjects for whom the data collected at the different time points could be merged. In addition, only subjects with valid values for all of the aforementioned variables are taken into account. The resulting sample includes 229 participants.

Table 1 shows that no striking initial differences can be found between members of the control (n = 57) and the treatment group (n = 172), hereby analyzed as a whole. Moreover, participants in the control condition (M = 45.8, SD = 31.8) and the treatment condition (M = 45.0, SD = 32.6) reported, on average, similar private opinions towards the target item statement. These findings are consistently supported by both the corresponding two-sample tests of proportions as well as the two-sample t-tests. None of the test results indicate statistically significant differences between both the main groups of participants at the outset of the study.

Table 1

Descriptive statistics (control vs. treatment group, t1).

Variables	Control Group M(SD)	Treatment Group M(SD)
Target Item (0–100)
“The killing and eating of […].”	45.8(31.8)	45.0(32.6)
Big Five Inventory (1–5)
Agreeableness	3.2(.8)	3.1(.8)
Conscientiousness	3.6(.8)	3.6(.8)
Extraversion	3.1(1.0)	3.3(.9)
Neuroticism	3.1(.9)	3.0(.9)
Openness	4.0(.8)	3.9(.9)
Covariates
Age in years	26.4(6.6)	27.4(7.2)
Vegetarians/Vegans	24.6%	20.9%
Students	80.7%	73.8%
Males	33.3%	31.4%
	n = 57	n = 172

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations.

The histograms and kernel density estimations presented in Fig 2, however, allow for a more detailed look at the initial opinion distributions, while also providing a first impression of the general outcome of the social feedback process that took place in the laboratory: Both groups not only show almost identical mean values at t1, but very similar median values as well. Thus, around half of the control group members reported a rather disagreeing private opinion (Mdn = 49, IQR = 71−15), the other half positioned themselves on the part of the scale that represents agreement. While the same is true for the treatment group members (Mdn = 48, IQR = 71−14.5), small differences in the opinion distributions are discernible as participants in the control condition were slightly more likely to take an either neutral stance or lean towards complete agreement; similarity of initial opinion distributions is nonetheless supported by the curves of the kernel density estimations.

Fig 2

Target item distributions (control vs. treatment group).

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations.

Fig 2 moreover illustrates that in the control group the distribution of our target item variable changed slightly, but insubstantially over the course of the study. For members of the treatment group, however, the opinion distribution was subject to a more pronounced change, which is in line with expectations. Conducting a paired-samples t-test to compare the opinions reported before (M = 45.0, SD = 32.6) and after (M = 40.3, SD = 35.4) the treatment reveals a small yet statistically significant reduction in average agreement, t(171) = −2.49, p =.014, d = −.19. A Wilcoxon signed-ranks test furthermore confirms the observable shift in the median value, from Mdn = 48.0 (IQR = 71−14.5) to Mdn = 27.5 (IQR = 77−7), to be statistically significant as well, Z = −2.28, p =.023, r = −.12. In fact, the majority of the treated reported disagreeing positions after participating in a laboratory session. The posttest histograms as well as the kernel density estimations also illustrate that while members of the control group were more prone to report less extreme positions at t3, there is a noticeable reduction of neutral opinions and a strengthening of the extreme ends of the scale for members of the treatment group after social feedback was applied, a tendency which has led to a slightly bi-polarized opinion distribution.

Assessing the actual impact of the various social feedback conditions on the private opinion of interest requires a closer look at the differences in intra-group changes, as provided in Table 2 and Fig 3: The reduction in average agreement, from M = 49.2 (SD = 33.4) at t1 to M = 46.4 (SD = 35.5) at t3, is smallest (d = −.10) and insignificant for those who received exclusively positive social feedback during a laboratory session. The same is true for the nonetheless notable shift in the group-specific median value (from Mdn = 57.0, IQR = 73−23 to Mdn = 42.0, IQR = 77−15). With a decrease from M = 53.9 (SD = 32.0) to M = 50.3 (SD = 35.7) the change of opinion is only somewhat more pronounced in the negative feedback condition (d = −.15). A comparison of the opinion distributions, as well as their differences after the respective social feedback was applied, in fact reveals surprising similarities between these two treatment groups. In addition to the similar shapes of the initial curves, comparably formed distributions of the private opinion are also to be found in the posttest data. Contrary to the theoretical expectations presented above, however, the impact of the laboratory treatment is strongest for participants in the mixed feedback condition. In this group the reduction in average agreement, from M = 37.0 (SD = 31.0) to M = 30.8 (SD = 33.1), is found to reach statistical significance, t(80) = −2.38, p =.020. Although the effect size is still rather small (d = −.26), the simultaneous exposure to the two opposing types of social feedback, i.e. positive and negative, has led many of the group members to express strong or even complete disagreement. This is also reflected in the median value, which decreased from Mdn = 31.0 (IQR = 66−8) to Mdn = 14.0 (IQR = 61−2) and thus supports the impression of the mixed feedback condition to be the most influential of the three, Z = −2.138, p =.033, r = −.17. Overall, it is striking that across all treatment groups an erosion of neutral positions took place. The application of a social feedback treatment in the laboratory therefore generally led more participants to express more pronounced positions at t3.

Table 2

Mean values (control vs. treatment groups, t1 vs. t3).

	n	t1 M(SD)	t3 M(SD)	Δ M(SD)	95% CI	Cohen’s dz
Control Group	57	45.8(31.8)	46.1(29.6)	.3(22.9)	[−5.8, 6.4]	.01
Treatment Group	172	45.0(32.6)	40.3(35.4)	−4.7(24.7)	[−8.4, −1.0]	−.19
Positive Feedback	37	49.2(33.4)	46.4(35.5)	−2.9(28.3)	[−12.3, 6.6]	−.10
Negative Feedback	54	53.9(32.0)	50.3(35.7)	−3.6(24.0)	[−10.2, 3.0]	−.15
Mixed Feedback	81	37.0(31.0)	30.8(33.1)	−6.3(23.6)	[−11.5, −1.0]	−.26

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations.

Fig 3

Target item distributions (treatment groups).

Source: ODSF2018, own calculations.

We thus observe that although the analyses of group-specific means point to only minor changes in the average opinions of the participants, a closer look at the actual opinion distributions reveals quite broad transitions at the individual level. This becomes even more obvious when further measures and representations are taken into account: The high standard deviations of the respective mean value differences in Table 2 as well as the box plots, especially of the differences, as presented in Fig 4, illustrate the extent to which participants’ individual opinions varied over the course of the study. Ultimately, since the means of the group-specific absolute differences, ranging from M = 15.1 (SD = 19.2) to M = 18.2 (SD = 21.6), are considerably large and quite similar across the four groups under study, the general extent of opinion changes on the individual level seems far from negligible. Therefore, we conclude that there was in fact a quite pronounced opinion volatility in all conditions.

Fig 4

Boxplots of target item distributions and differences.

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations.

Moreover, a comparison of the various treatment groups with respect to the initial levels of both the target item variable and the covariates indicates that, despite extensive randomization and the seemingly sufficient equality of the control and treatment group members in general (Table 1), the compositions of the three treatment groups turn out to be different in some respects (see Table 3). While 40.7% of the negative feedback group members identified themselves as male, this is true for the smaller share of 25.9% of the participants in the mixed feedback condition. This difference, however, is not statistically significant at the 5% level. On the other hand, 80.2% of the latter were students at the time of study participation, in contrast to only 63.0% of the subjects in the negative feedback group, z = −2.22, p =.026, that also shows to be statistically significantly different from the control group (80.7%), z = 2.08, p =.037. The members of the mixed feedback group agreed significantly less with the target item statement at the beginning of the study than did the individuals in the negative feedback group, t(113) = 3.03, p =.003. In addition, a statistically significant difference between the control and the positive feedback group was found for the personality trait of extraversion, t(82) = −2.02, p =.047. This suggests that the second randomization step did not lead to almost identical groups.

Table 3

Descriptive statistics (control vs. treatment groups, t1).

Variables	Control M(SD)	Positive M(SD)	Negative M(SD)	Mixed M(SD)
Target Item (0–100)
“The killing and eating of […].”	45.8(31.8)	49.2(33.4)	53.9(32.0)	37.0(31.0)
Big Five Inventory (1–5)
Agreeableness	3.2(.8)	2.9(.8)	3.0(.8)	3.2(.7)
Conscientiousness	3.6(.8)	3.8(.8)	3.6(.8)	3.5(.8)
Extraversion	3.1(1.0)	3.5(.9)	3.3(.9)	3.2(.9)
Neuroticism	3.1(.9)	2.9(1.1)	2.9(1.0)	3.2(.8)
Openness	4.0(.8)	3.9(1.0)	3.8(1.0)	3.8(.9)
Covariates
Age	26.4(6.6)	26.1(6.3)	29.3(9.4)	26.8(5.7)
Vegetarians/Vegans	24.6%	18.9%	14.8%	25.9%
Students	80.7%	75.7%	63.0%	80.2%
Males	33.3%	29.7%	40.7%	25.9%
	n = 57	n = 37	n = 54	n = 81

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations.

Linear mixed-effects regression

As a result of the pretest-posttest-design, the opinions of each participant collected at different times are represented by two data points per item. Assuming that unobserved subject-specific characteristics existed that remained constant throughout the study period, these values cannot be considered as independent observations. Moreover, actual differences in group compositions, as identified in the previous section, argue against causal inferences solely based on direct comparisons of the various groups. And in addition, participants’ self-assignment to the various laboratory sessions made it likely that subjects who chose to participate in a specific session, as for instance on a Saturday morning, also might have shared unknown and unobserved characteristics. Ultimately, hypotheses H1 and H2 imply that the direction of a potential social feedback treatment effect depends on the initial position of the person whose opinion is subject to judgment. It is therefore appropriate to separately analyze the two fundamental baseline conditions in which either a disagreeing private opinion was held at t1 or an agreeing position existed.

In order to account for clustering on the session-level, repeated measurements on the subject-level, differences in group composition, and the directional implications of the hypotheses we estimated three multiple three-level linear mixed-effects regression models on the subjects’ opinion regarding the target item statement. While model A incorporates those participants who submitted a disagreeing private opinion (, n = 120) towards the target item statement at the beginning of the study, model B consists of participants which reported an agreeing initial position (, n = 109). Both the models A and B are of particular relevance for the evaluation of our hypotheses and provided in full in Table 5 alongside a third model C, which incorporates the entire sample (n = 229).

In each of the models the laboratory sessions represent the top level (level 3 groups). Repeated measurements of the target opinion on the lowest level (level 1 observations) are considered to be nested within subjects (level 2 groups). Predictor variables and covariates are included as fixed effects, with the former being binary variables indicating the specific experimental group membership. As a consequence, the associated partial regression coefficients can be interpreted in contrast to the respective groups’ state at t1. The Big Five personality traits, vegetarian or vegan diet, student status, sex, and age are included as covariates; BFI-10 variables and age are z-transformed. After converting the data from wide to long format, the dependent variable is composed of the individuals’ target item measurements at times t1 and t3. For each participant it thus contains two integer values ranging from 0 (complete disagreement) to 100 (complete agreement), the first one resulting from the pretest at t1 and the second measured during the posttest at t3. Table 4 illustrates the structure of the dataset used for the multilevel analysis presented within this section. Unobserved heterogeneity on the session-level and unobserved differences between subjects are controlled by specifying respective random intercepts. In addition to that, each participant is assumed to have a subject-specific random slope, hence allowing for varying treatment effects among subjects. In general, the corresponding likelihood-ratio tests show that introducing random intercepts on the session- and subject-level led to models that are statistically significantly superior to their linear equivalents, which are blind to the specifics of our multilevel data structure.

Table 4

Illustration of the dataset (long format).

ID	Time of Observation	Dependent Variable	Control Group	Positive Feedback	Negative Feedback	Mixed Feedback	Age (z-Score)	…
1	0	50	0	0	0	0	-.58	…
1	1	47	1	0	0	0	-.58	…
2	0	40	0	0	0	0	-.44	…
2	1	100	0	1	0	0	-.44	…
3	0	27	0	0	0	0	-.72	…
3	1	22	0	0	0	1	-.72	…
…	…	…	…	…	…	…	…	…

Fixed effects

A first look at the fixed-effects part of Table 5 shows that both models have differing overall-intercepts, which naturally results from separating the two specific sample cutouts of interest. For both the initially disagreeing participants (estimate = 21.06, p <.001, 95% CI [13.56, 28.55]) and the sample of individuals formerly found on agreeing positions (estimate = 73.88, p <.001, 95% CI [66.34, 81.41]), intercepts are estimated to be close to the middle of their respective part of the scale.

Table 5

Results of the multiple three-level linear mixed-effects regressions on the subjects’ opinion towards the target item statement.

	Model A (y_i_t_1<50)	Model B (y_i_t_1≥50)	Model C (full sample)
Random Effects	SD (SE)	SD (SE)	SD (SE)
Session Intercept	4.13***(2.65)	.00(.)	.00(.)
Subject Intercept	13.21***(1.62)	13.95***(1.68)	24.67***(1.50)
Social Feedback Treatment
Subject: Control Group	17.40***(4.06)	16.03***(3.67)	.00(.00)
Subject: Positive Feedback	28.39***(6.32)	20.72***(4.30)	13.77***(6.04)
Subject: Negative Feedback	16.01***(4.14)	23.96 ***(3.72)	9.24***(6.35)
Subject: Mixed Feedback	10.08***(3.45)	29.14***(1.68)	4.15***(11.53)
Fixed Effects	Estimate (SE)	Estimate (SE)	Estimate (SE)
Intercept	21.06***(3.82)	73.88***(3.84)	49.55***(4.58)
Social Feedback Treatment
Control Group	7.51+(3.93)	−7.29*(3.68)	1.14(2.89)
Positive Feedback	5.69(7.78)	−10.01*(4.99)	−1.96(4.22)
Negative Feedback	3.88(4.47)	−8.21+(4.50)	−2.17(3.23)
Mixed Feedback	−1.08(2.30)	−16.14**(5.71)	−8.22**(2.49)
Big Five Inventory
Agreeableness†	−1.26(1.53)	−2.48(1.58)	−5.33**(1.87)
Conscientiousness†	−2.27(1.43)	−.45(1.68)	−1.91(1.86)
Extraversion†	1.15(1.72)	1.73(1.59)	0.05(2.00)
Neuroticism†	−.53(1.67)	−.89(1.66)	−1.19(2.03)
Openness†	−.73(1.75)	−1.50(1.47)	−3.21+(1.94)
Diet
Neither Vegetarian nor Vegan
Vegetarian or Vegan	−10.60**(3.19)	−12.29*(5.38)	−27.98***(4.49)
Student Status
Not a Student
Student	.32(4.18)	.46(4.44)	.88(5.24)
Sex
Female
Male	1.78(3.54)	2.27(3.46)	3.33(4.27)
Age†	.13(2.11)	.88(1.69)	2.35(2.24)
AIC	2048.9	1888.8	4251.0
χ2 (LR Test vs. Linear Model)	48.09	56.52	133.75
n	120	109	229

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations. Notes: N observations = 458, N subjects = 229, N sessions = 19, standard errors in parentheses, † z-transformed, reference categories in italics, + p ≤.10, * p ≤.05, ** p ≤.01, *** p ≤.001 (two-tailed). The restricted maximum likelihood estimations (REML) were performed using the mixed command as provided in Stata 15.1.

There is a fairly clear movement of private opinion towards the neutral middle of the full target item scale for the untreated control group in both models. As can be seen in Table 5 and Fig 5, the agreement with the target item statement increased, on average, by 7.51 (p =.056, 95% CI [−.20, 15.22]), when there was a disagreeing private opinion at the beginning and no social feedback was applied, whereas in model B the agreement decreased to a similar extent (estimate = −7.29, p =.047, 95% CI [−14.50, −.08]). In general, it can be observed that although the tendency towards the neutral positions is evident in almost all of the treatment groups, it’s extent varies depending on the respective feedback condition. Moreover, social feedback generally resulted in formerly disagreeing subjects tending somewhat less towards the center of the original target item scale than is the case for the treated in model B.

Fig 5

Coefficients plot of the multiple three-level linear mixed-effects regression.

Source: ODSF2018 (https://doi.org/10.6084/m9.figshare.16595378.v1), own calculations. Notes: N observations = 458, N subjects = 229, N sessions = 19. BFI-10 indices and age are included as z-transformed variables, intercepts are omitted. The graph was generated using the coefplot package for Stata [34].

For both, positive (estimate = 5.69, p =.465, 95% CI [−9.56, 20.945]) and negative social feedback (estimate = 3.88, p =.386, 95% CI [−4.89, 12.65]), the tendency towards the center is slightly reduced in model A. With an estimated average reduction in agreement of −1.08 (p =.638, 95% CI [−5.59, 3.42]), the comparably strongest influence on the change of opinion once again can be found for the mixed feedback condition. In the sample of those who gave an agreeing private opinion at t1, a striving towards the neutral part of the target item scale is also evident. However, while in model A social feedback slows down the movement towards the scale center and binds the participants more strongly to the disagreeing areas, in model B the application of social feedback has had a slightly catalyzing effect that drives the movement towards the less agreeing parts of the scale. The different treatment groups thus tend to show slightly more pronounced decreases, with the most prominent effect again in the mixed feedback group (estimate = −16.14, p =.005, 95% CI [−27.33, −4.96]). When comparing the confidence intervals of the group-specific changes in private opinion over time (Fig 5), though, despite the observable trends neither model A nor B show statistically significant differences between the various groups.

Random effects

For individuals in the initially disagreeing subsample, the standard deviation of the random session-intercept (SD = 4.13, 95% CI [1.18, 14.51]) indicates statistically significant variations between baseline levels in the various laboratory sessions. This cannot be found for the originally agreeing subjects. However, random intercepts show quite strong and statistically significant variability at the subject-level in model A (SD = 13.21, 95% CI [10.38, 16.80]) as well as in model B (SD = 13.95, 95% CI [11.01, 17.67]). In all groups a strong between-subjects variation of the slopes is apparent. Thereby, its extent is similar between control group members in both models, with SD = 17.40 (95% CI [11.01, 27.50]) in model A and SD = 16.03 (95% CI [10.24, 25.10]) in model B. In contrast, a different picture emerges for the treatment groups: Whereas in model A the between-subjects variability of the slopes is highest in the positive feedback group (SD = 28.39, 95% CI [18.35, 43.92]) and lowest in the mixed feedback group (SD = 10.08, 95% CI [5.16, 19.71]), things turn out the other way around in model B, in which we estimate SD = 20.72 (95% CI [13.79, 31.13]) for the former and SD = 29.14 (95% CI [21.57, 39.37]) for the latter.

Discussion

The question of how and to what extent people influence each other in social exchange processes has been of interest for a long time. It is of growing relevance as large segments of people nowadays are involved in online communication and interaction settings in which they find themselves exposed to social judgments by unknown others.

While the debate about potential implications of these novel exchange processes is carried out with fervor, empirical evidence is scarce. This paper is contributing to this discussion with an experimental study that analyzed the influence of social feedback on actual private opinions. Our research questions and hypotheses were inspired by social feedback theory [18] which centers around George Homans’ idea that human beings adapt their behavior according to the reactions it evokes from others [3].

We derived and empirically tested three hypotheses. H1 asserts that social feedback perceived as positive causes recipients to push their private opinion further in the direction of the original position, whereas H2 states that people who receive negative social feedback will strive towards the opposite direction of their initial opinion. We furthermore expected (H3) that the administration of both positive and negative feedback would result in no change in the private opinion of interest. The hypotheses were tested through an experimental design that combined a preceding and a follow-up online survey with a laboratory session in which participants were asked to evaluate a statement regarding meat consumption and subsequently received a social feedback stimulus of either positive, negative or mixed content.

The descriptive analyses show that both the control and the treatment group in general shared very similar target opinion distributions at the outset; with each half of the participants reporting either rather agreeing and disagreeing opinions. Following the social feedback treatment, there was only a small negative effect on the average agreement with the target item statement in both the positive and the negative feedback group. Contrary to our expectation, however, a statistically significant reduction in agreement occurred among subjects that got randomly assigned to the mixed feedback treatment. Besides, it is striking that we observe an erosion of neutral positions in all three social feedback conditions which leads to opinion distributions that are notably bi-polarized.

In order to test our hypotheses according to the requirements of our data, we estimated several multiple three-level linear mixed-effects regression models. For the control group it becomes apparent that participants who originally disagreed with the target item as well as those who initially agreed moved to an almost equal extent towards the respective opposite side of the scale. In principle, this also applies to participants who received either unambiguous positive or negative social feedback. If an equal mixture of positive and negative feedback statements is applied, however, it is predicted that the natural striving to the middle of the scale will be affected: While initially disagreeing individuals are predicted to hold on to their disagreeing stance, it is estimated that individuals that agreed in the beginning move even further towards the less agreeing domains than subjects in other feedback conditions.

In principle, both research questions, namely whether social feedback is influential and whether this influence depends on the type of feedback, can be answered affirmatively. However, the specific effects as claimed in the hypotheses could not be confirmed. Our findings are nonetheless remarkable exactly because they contradict the theoretical expectations that are plausible and obvious at first glance. This is particularly evident in light of the fact that it was not merely spontaneous public adjustments that had been observed, but actual and persistent changes in private opinions measured several days to weeks after the social feedback was applied. It is all the more astonishing as participants were led to believe that they had received this feedback within the context of a singular interaction with unknown and unidentifiable strangers in a sanction free lab environment. These unexpected initial results motivate further investigations as well as replication studies. This is also demanded by the marked degree of opinion volatility of which only a small proportion can be attributed to the social feedback alone. Since private opinions can vary for manifold external and internal reasons, this may naturally result from the repeated measurement design spanning over several weeks. Overall, the participants’ opinions do not seem to have been especially stable and persistent, but have undergone variations based on presumably many additional unobserved factors.

For future studies, we suggest conceptual replications and the further development of the presented experiment in order to validate our findings. First, with regards to content, it would be important to extend the range of opinion statements that participants receive feedback for. In order to get a first and fundamental insight into the effect of social feedback on private opinions, we chose the alleged study object with caution, aiming for a topic that was neither strongly polarized nor unfamiliar to participants. We did not however measure the relative importance that the attitude towards the killing and eating of animals holds for participants which is one of the limitations of our study. With regards to the most urgent questions of opinion polarization and its potentially detrimental effects on social cohesion it would be important to test the effect of social feedback on convictions that are of crucial importance to participants’ identities and therefore emotionally charged. In line with the theoretical reasoning that we presented above we expect that even in this case exposure to social feedback will generally result in a change in opinion in the direction that participants believe to be rewarded by their social surroundings. Yet for this change to be distinct and lasting we expect that the social feedback stimulus would have to be implemented more frequently and by a larger number of feedback donors. It is also unclear whether the expected assimilation of these more relevant opinions would occur in a situation that is completely anonymous and sanction free. An experimental set-up which varies the degree of anonymity and potential punishment (as for example the prospect of face to face interaction in the future would provide) could shed some light on these questions which we hope will be addressed by future research. Secondly, and on a methodical note, we have to acknowledge that the slider used in the online questionnaires itself may have been a source of dispersion. In retrospect, it seems to be quite difficult to give the exact same opinion twice using such a sensitive response format that incorporates around 100 possible opinion values, even if the opinion has not actually changed. In this respect, some noise could have been introduced by the sliders alone and may already be eliminated by implementing a different response format.

We hope that our research contributes to the necessary debate on the influence of social feedback on opinion changes. As networks of information and opinion exchange are likely to gain even more density and importance in the years to come and as empirical evidence on the influence of social feedback on opinions is scarce, we believe that this field of research deserves more attention than it has received so far.

Social feedback statements.

List of all social feedback statements used during the laboratory treatment process.

(PDF)

Click here for additional data file.

Variable report.

Explanations of all variables in the dataset.

(PDF)

Click here for additional data file.

6 Jun 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Ning Du

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the Methods section, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

3. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

4. Please note that in order to use the direct billing option the corresponding author must be affiliated with the chosen institute. Please either amend your manuscript to change the affiliation or corresponding author, or email us at plosone@plos.org with a request to remove this option.

5. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

Additional Editor Comments (if provided):

This is an interesting topic and your finding of polarization of initial opinion is especially relevant in today’s environment. My major concern is your analysis, which should focus on the change/difference instead of the level.

1) In your linear models, you stated that your dependent variable is composed of the individuals’ target item measurements at times t1 and t3. Please clarify the specific measurement of your DV. Is it the difference between t1 and t3, the average, or the sum? Or is it the opinion at t3?

2) In Table 2 below, could you explain why at t1, the mixed feedback group is significantly lower than other groups? This will not change your results if the analysis focuses on the difference between t1 and t3. In addition, in this group, you have a much higher number of participants.

3) You provided means of the group-specific absolute differences (ranging from M = 15.1 (SD = 19.2) to M = 18.2 (SD = 21.6), are considerably large and quite similar across the four groups). Please explain the significance of this measurement. I assume it is the absolute difference at the individual level for each subject.I think this measurement is superior to the regular difference if you follow up with more analysis. For example, you can break it down in 3 groups, 1) more positive opinion after t2, t3-t1>0, 2) more negative, t3-t1<0, 3) no change, t3-t1=0. This will not only show the direction but also the magnitude of the change.

4) You find the target item variable and the covariates significantly different despite “seemingly sufficient equality of the control and treatment group members”. You should include analysis to make sure whether the means are statistically different. In your Linear Mixed-Effects Regression, also think about including these covariates as control variables.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting study of the effects (if any) of social feedback on attitude change. The experiment is well-designed and the results are written up clearly, for the most part.

Small thing I would like to see:

More discussion of the scatterplots (Fig 5), especially interpretation of their meaning. I have not seen this particular form of visualization in the context of experiment-to-experiment change, and it was a bit confusing at times.

Some of the language of results could be cleaned up. For example, on Page 13 authors state that "Once again it is the mixed feedback condition 441 in which the most striking effect is seen, with an estimated average reduction in 442 agreement of −1.08." While I parsed apart that what the authors (I think) were referring to is an interesting *non* effect, stating that an effect exists in the presence of a non-effect is confusing. I would rephrase this.

Bigger things:

The authors do well to select an issue that, while contentious (humans eating meat), is not polarized in terms of ideology and partisanship in the European context. Obviously an issue cueing ideology (say, EU membership issues) would change the design quite a lot. However, I would like to see discussion of what the expectation would be in that context, at least in the discussion, given that much of the manuscript is motivated by increasing opinion polarization (supposedly) driven by online communicative domains.

How does motivated reasoning fit into this theoretical framework? If people have strongly-held attitudes on this (or any) issue going into the experiment, that is going to dramatically alter how they perceive positive or negative feedback dramatically, I would think. For example, if I say that humans are meant to eat meat at 90% on the temperature scale, my response to negative feedback might be very different than someone who goes into the lab at 60%, or 40%. Discussion of how dissonance induced by a countervailing attitude is necessary.

Moreover, I think this would be able to be analyzed given the data at hand - I would like the authors to address this by examining not only movement in opinions in the face of pro-/counterattitudinal social cues, but also how those moves (if any) are influenced by how strongly-held those opinions are at the outset. I could imagine some of this being taken care of by presenting plots of random slopes, split (by graph) across participants with strongly- versus weakly-held attitudes.

The Big 5: Why? The personality battery first shows up in the descriptive results (Page 13 of the reviewer copy) but no discussion is made prior to this, or even during the analysis, as to why we would expect the Big Five to be important to control for. More discussion is needed on this point.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

5 Sep 2022

(1) In your linear models, you stated that your dependent variable is composed of the individuals’ target item measurements at times t1 and t3. Please clarify the specific measurement of your DV. Is it the difference between t1 and t3, the average, or the sum? Or is it the opinion at t3?

Response: Thanks for pointing out that the description of the dependent variable is not clear. We have made changes accordingly. The analysis focuses on the differences between t1 and t3. As you correctly note, the dependent variable is composed by the repeated target item measurements at time points t1 and t3, i.e. before as well as after the lab sessions. In order to account for the hierarchical structure of the data, the data set has been put into long format, so that instead of having one variable per time point of measurement, a single dependent variable contains the two measurements of opinion for each participant. Thus, we can observe the inter-group differences in the intra-group changes of opinion. In the manuscript, the specifics of the data set structure are now illustrated in Table 4.

(2) In Table 2 below, could you explain why at t1, the mixed feedback group is significantly lower than other groups? This will not change your results if the analysis fo-cuses on the difference between t1 and t3. In addition, in this group, you have a much higher number of participants.

Response: Unfortunately, we cannot explain why the mixed feedback group is composed of participants that on average showed a significantly lower agreement at the beginning of the study. The specific treatment group membership was manifested in the laboratory when participants were randomly assigned their feedback statements (this was the second ran-domization step). A subsequent check of the z-Tree code showed that the random assignment of feedback statements was flawless and indeed purely random. Thus, the difference between the mixed feedback group and the other groups has to be considered a product of coincidence.

The random assignment of social feedback statements initially resulted in four groups: One group of individuals who received exclusively positive feedback, one with exclusively negative feedback, one in which the fictitious group member A’s statement was positive und C’s was negative, and one group in which A appeared to give negative feedback and C positive feedback. Thus, randomization initially created four groups of approximately equal size.

As a first descriptive analysis showed, the two mixed feedback groups are composed in a very similar way of people who had a lower average agreement with the target item at the beginning of the study – compared to the people in the positive and negative feedback groups; overall, the mixed feedback groups showed to be similar to each other. Furthermore, we have no theoretical assumptions about how the effect of social feedback differs when the order of feedback statements presented simultaneously on the same page is reversed. For these reasons, we decided to group all individuals who received a positive and a negative feedback statement into a single group, the mixed feedback group. This led the latter to be larger in size than the positive and the negative feedback group.

In general, differences in group sizes also arose, for example, because participants did not take part in the last survey or entered their ID codes incorrectly, so that it was not possible to merge their data properly.

(3) You provided means of the group-specific absolute differences (ranging from M = 15.1 (SD = 19.2) to M = 18.2 (SD = 21.6), are considerably large and quite similar across the four groups). Please explain the significance of this measurement. I assume it is the absolute difference at the individual level for each subject. I think this measurement is superior to the regular difference if you follow up with more analysis. For example, you can break it down in 3 groups, 1) more positive opinion after t2, t3-t1>0, 2) more negative, t3-t1<0, 3) no change, t3-t1=0. This will not only show the direction but also the magnitude of the change.

Response: By reporting the average group-specific absolute differences, we want to point out the phenomenon that considering only the average group-specific differences – as provided in Table 2 – can lead to an underestimation of the extent of opinion volatility at the individual level. For example, for the control group (M_t1= 45.8 and M_t3 = 46.1), there is such a small change in average agreement that one could think that the participants’ opinions remained almost unchanged over the course of the study. However, some control group members reduced their agreement, while others increased their agreement to a comparable extent. Thus, the aggregation, i.e. the mean value, creates the impression that the opinions had hardly changed, while in fact there are significant differences at the individual level.

The average absolute differences provide us with the means to show the extent to which, on average, there was a change of opinion for each person in a group, regardless of its direction. This is to show that, in general, a comparably high opinion volatility can be found in all groups studied; also in the control group, which was not exposed to any social feedback. The difference between the groups, however, can be seen in the fact that the directions of individual changes of opinion had been influenced by social feedback – as in the case of the mixed feedback group, in which on average a reduction in agreement occurred, i.e. increases and decreases in agreement on the individual level did not cancel each other out, but rather the decreases outweighed the increases.

Since our hypotheses not only assert that social feedback leads to changes of opinion, but rather assume specific directions of change depending on the type of social feedback, we do not consider the absolute differences to be a suitable measure for assessing the validity of the hypotheses that are at the core of this study.

(4) You find the target item variable and the covariates significantly different despite “seemingly sufficient equality of the control and treatment group members”. You should include analysis to make sure whether the means are statistically different. In your Linear Mixed-Effects Regression, also think about including these covariates as control variables.

Response: Starting at line 322, we address the question of whether the control group and the treatment group, analyzed as a whole, differed from each other, or whether they actually did not show significantly different measures, as we expected based on the random assignment of our subjects. We found that there were no statistically significant differences between the control and treatment group with respect to the target item, the Big Five Inventory, and the control variables age, diet, student status, and gender (see Table 1). We did not report the insignificant test results in order not to make the text too confusing.

Starting at line 397, we address the phenomenon that despite randomized assignment of social feedback statements, the various treatment groups differ from each other in certain respects. Once again, we thank you for the important comment. In fact, we’ve missed to report the respective test results. Statistically significant differences are now reported in the manuscript.

All of the aforementioned covariates had been included as control variables in all of the linear mixed-effects models, see Table 5 as well as Fig 5.

(5) More discussion of the scatterplots (Fig 5), especially interpretation of their meaning. I have not seen this particular form of visualization in the context of experiment-to-experiment change, and it was a bit confusing at times.

Response: The discussion of the mean values of the absolute differences and their graphical representation in Fig 4 were intended to emphasize the fact that, overall, a high degree of opinion volatility was found. This was also to be underlined by the scatterplot, which visualizes all actual individual changes of the target opinion over time and reveals that clear patterns of change can hardly be identified. We agree that this is perhaps too much fuss over a less informative fact and have thus decided to remove the scatterplot in order to avoid confusing redundancy.

(6) Some of the language of results could be cleaned up. For example, on Page 13 authors state that “Once again it is the mixed feedback condition 441 in which the most striking effect is seen, with an estimated average reduction in 442 agreement of −1.08.” While I parsed apart that what the authors (I think) were referring to is an interesting *non* effect, stating that an effect exists in the presence of a non-effect is confusing. I would rephrase this.

Response: Thanks for pointing this out, we’ve changed the wording accordingly.

(7) The authors do well to select an issue that, while contentious (humans eating meat), is not polarized in terms of ideology and partisanship in the European context. Obviously an issue cueing ideology (say, EU membership issues) would change the design quite a lot. However, I would like to see discussion of what the expectation would be in that context, at least in the discussion, given that much of the manuscript is motivated by increasing opinion polarization (supposedly) driven by online communicative domains.

Response: Thank you for giving us the opportunity to elaborate on this point. It is without doubt necessary that a validation of our design and findings includes topics that are linked to participants’ identities more closely and therefore more emotionally charged. As our study is (to our knowledge) the first laboratory study which focusses on measuring the direct mechanism that lead people to changing their private opinion after being presented with anonymous social feedback, we wanted to start with a topic that was neither polarized nor unknown to participants and wouldn’t activate other mechanisms (e.g. Backfire effect). We do believe that integrating a topic that is more directly linked to participants´ identities and therefore of greater importance to them would still yield the same effects in the long run, but that the stimulus in this case would have to be more frequent and the number of participant donors would have to be greater. It is a point that we will consider in our future research. In pointing out these assumptions we have included the following paragraph in the discussion of our manuscript.

(8) 4. How does motivated reasoning fit into this theoretical framework? If people have strongly-held attitudes on this (or any) issue going into the experiment, that is going to dramatically alter how they perceive positive or negative feedback dramatically, I would think. For example, if I say that humans are meant to eat meat at 90% on the temperature scale, my response to negative feedback might be very different than someone who goes into the lab at 60%, or 40%. Discussion of how dissonance induced by a countervailing attitude is necessary.

Response: This comment sparked some discussion for future research in our group as we unfortunately did not include a measure of attitude importance in the present study.

Assuming that a high value on the attitude scale (indicating agreement or disagreement) equals the importance that the opinion has for the respective person would in our opinion mix up different aspects of opinion that should be treated separately. Just because a person is in complete disagreement with the target statement does not indicate the centrality of the topic. We therefore do not see the opportunity to make testable assumptions about this point on basis of our study design and data base.

We have updated the manuscript by naming the lack of an appropriate measure of attitude importance as one of the limitations of our study

(9) Moreover, I think this would be able to be analyzed given the data at hand - I would like the authors to address this by examining not only movement in opinions in the face of pro-/counterattitudinal social cues, but also how those moves (if any) are influenced by how strongly-held those opinions are at the outset. I could imagine some of this being taken care of by presenting plots of random slopes, split (by graph) across participants with strongly- versus weakly-held attitudes.

Response: As stated above we do not think that we have the necessary information for making assumptions that include attitude importance. One idea that came to mind in order to make use of what we do have, was to use diet as a proxy of attitude importance, yet our participant groups in that case would be too small to allow for statistically sound testing. This certainly marks a limitation of our study and should be addressed in the future.

(10) The Big 5: Why? The personality battery first shows up in the descriptive results (Page 13 of the reviewer copy) but no discussion is made prior to this, or even during the analysis, as to why we would expect the Big Five to be important to control for. More discussion is needed on this point.

Response: We agree that our decision to include the Big Five Inventory was lacking context in the first draft. As we point out in the updated version of our manuscript, including the Items of the Big Five Inventory Short Scale is our attempt to control for potentially unobserved participant characteristics that would systematically distort the influence of social feedback on private opinions. As we are covering new ground with regards to sociological research, we wanted to make sure not to miss out on nuances that could bias our results.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

7 Sep 2022

The effects of social feedback on private opinions. Empirical evidence from the laboratory.

PONE-D-21-30777R1

Dear Dr. Sarközi,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ning Du

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

26 Sep 2022

PONE-D-21-30777R1

The effects of social feedback on private opinions. Empirical evidence from the laboratory.

Dear Dr. Sarközi:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Ning Du

Academic Editor

PLOS ONE