Effects of restricting social media usage on wellbeing and performance: A randomized control trial among students.

Recent research has shown that social media services create large consumer surplus. Despite their positive impact on economic welfare, concerns are raised about the negative association between social media usage and well-being or performance. However, causal empirical evidence is still scarce. To address this research gap, we conduct a randomized controlled trial among students in which we track participants' daily digital activities over the course of three quarters of an academic year. In the experiment, we randomly allocate half of the sample to a treatment condition in which social media usage (Facebook, Instagram, and Snapchat) is restricted to a maximum of 10 minutes per day. We find that participants in the treatment group substitute social media for instant messaging and do not decrease their total time spent on digital devices. Contrary to findings from previous correlational studies, we do not find any significant impact of social media usage as it was defined in our study on well-being and academic success. Our results also suggest that antitrust authorities should consider instant messaging and social media services as direct competitors before approving acquisitions.

Introduction

Social media increasingly plays an important role in our daily lives. Ever since the launch of major modern social media platforms such as Facebook, users have adopted them at an explosive pace and adoption continues to increase to this day. Almost 3 billion users are active monthly on Facebook in 2021 [1]. This corresponds to over a third of the global population. The average adult spends over 45 minutes every day on social media platforms [2].

Given this rapid adoption and usage of social media platforms, it is essential to study the impact of social media on the well-being of users. Brynjolfsson et al. [3] find that digital technologies, including social media, generate a large amount of consumer surplus. More specifically, they conduct incentive compatible choice experiments to measure the consumer surplus generated by Facebook and find that the median US Facebook user obtains around $48/month of value from using Facebook in 2017 as measured from their willingness to accept to give up access to Facebook for a month. They also conduct a similar experiment with students at a large European university and find that the median student in their sample obtains €97/month of value from using Facebook.

While Facebook and other social media services seem to generate a large amount of consumer surplus and contribute towards the economic well-being of their users, questions are raised about the negative externalities generated by social media. There is an active debate in media and academic research about the impact of social media on subjective well-being (including happiness and life satisfaction) and productivity. Current empirical results are ambiguous. Across different studies, correlational evidence points towards a positive, neutral (null results) and negative relationship between social media use and well-being (see Haidt [4] for a comprehensive literature review of social media use and mental health). However, most of this evidence suffers from issues related to reverse causality [5] and inaccurate measures of self-reported social media use [6]. Rigorous causal evidence on long term impacts of social media use on well-being is lacking.

Concerns are also raised in the field of Education policy on the impact of screen time (including social media use) on academic performance of students. Critics contend that social media use on smartphones distracts students from focusing in classes and affects their grades. Motivated by these concerns, the French education ministry banned smartphones in schools from first through ninth grades [7]. The American Academy of Pediatrics also recommends parents to limit the time spent by children and adolescents on social media so that they have enough time left to study [8]. However, a rigorous analysis of the data used in previous correlational studies that were used as evidence to support these policies suggests that the effects of social media use and screen time on adolescent well-being are too small to warrant policy changes [9].

Given these widespread concerns and conflicting correlational evidence on the impact of social media on well-being, it is necessary to obtain causal evidence in a timely manner before policies are implemented hastily. We seek to fill this research gap by conducting a randomized controlled trial to measure the causal long-term impact of social media use on academic performance and well-being. Specifically, the research question we aim to answer is: How does restricting social media use affect a) well-being, b) academic performance, and c) other digital activities?

We recruit students at a large European university to be part of our study over the course of three academic terms (quarters). The subjects install a software (RescueTime) on their personal computers and mobile devices. This software tracks all of the digital activities of the subjects during the entire duration of the study period. The first term serves as the baseline period. In the second term, subjects are randomized into treatment and control groups and the treatment group has social media use (Facebook, Instagram and Snapchat) restricted to a maximum of 10 minutes per day across all devices. We then measure the post-treatment effects in the third term.

We observe the entire space of digital activities performed by our subjects that covers online and also offline activities on their devices, including activities related to learning (such as writing in Microsoft Word or reading a PDF). Our social media use metrics are computed based on the actual time spent on social media and are not based on self-reported metrics of time spent, which is predominantly used in the existing literature. In addition to the digital activities, we obtain objective metrics of performance (grades) and subjective well-being scores solicited through surveys.

Contrary to results from previous studies using observational data, we do not find evidence that social media causes a positive or negative impact on well-being (including life satisfaction and mental health). Moreover, we also do not find any evidence that social media usage impacts academic success. However, we find significant substitution effects. Specifically, we see that participants in the treatment group substituted their use of social media services for instant messaging apps (e.g. WhatsApp). In total, these participants do not spend less time on their digital devices (computers and mobile phones) as those in the control group.

Our paper makes three main contributions. First, we test the popular media narrative portraying social media as the villain responsible for negatively affecting well-being of society. We do not find any evidence supporting this hypothesis. Second, educators and parents are increasingly concerned about the impact of digital distractions on academic performance and are restricting the online activities of students (for example through parental control software or by taking away their devices). While previous evidence seems to suggest that device usage in class might negatively affect academic performance, our results show that restricting social media usage from the lives of students (inside and outside class) might not have the intended effect. Finally, our paper is one of the first to provide evidence of substitutability between social media and instant messaging apps. This has major implications for antitrust authorities analyzing the market power of major social media platforms such as Facebook which owns Instagram (another social media service) and WhatsApp (instant messaging service).

The paper proceeds as follows. In the next section, we provide a brief review of existing literature on the impact of social media use on well-being and academic performance. In the following section, we describe the design of our experiment and data collected over the course of the study. We then show the main results and conclude with a discussion of the limitations of this study and directions for future research.

Related literature

The impact of the internet in general, and social media in particular, on well-being has attracted the attention of a number of researchers in the fields of psychology, epidemiology and human-computer interaction (HCI) over the past two decades. Most of this literature uses self-reported metrics of technology use and provides cross-sectional correlational evidence. Kraut and Burke [10] provide a review of this literature and express skepticism regarding cross-sectional and survey-based studies due to the presence of several confounding factors. Moreover, correlational studies might suffer from an abundance of researcher degrees of freedom and the file drawer problem such that only significant results are published, inevitably leading to the implication that social media usage either has a positive or negative effect. However, a null result or insignificant findings regarding social media usage might be a plausible outcome.

Orben and Przybylski [9] rigorously analyze popular large scale social datasets (n = 350k) used in previous correlational studies studying the impact of technology use on well-being by conducting a specification curve analysis of the data. This analysis involves running all possible analytical models using various combinations of the covariates. Instead of selective reporting, results from all of these analyses are reported. They find a small negative association between digital technology use and adolescent well-being. However this effect is economically insignificant explaining at most 0.4% of the variation in well-being. For comparison, the authors show that seemingly neutral activities such as eating potatoes have the same negative association with well-being as technology use. Given these concerns with correlational analyses involving cross sectional data, Kraut and Burke [10] call for experimental evidence paired with tracking data to provide reliable evidence on the relationship between internet use and well-being.

The subset of literature focusing on the association between social media use and well-being has found a wide range of effects (negative, mixed, positive and null). Using a longitudinal survey, Shakya and Christakis [11] found a negative association between Facebook use and well-being. In contrast, Burke et al. [12] find a positive association between directed communication on Facebook and social well-being due to subjects reporting improved feelings of social bonding and reduced loneliness. Similarly, Hobbs et al. [13] match Facebook profiles with public health records and find that being more socially integrated online (by accepting more Facebook friends) is associated with reduced risk of mortality. Burke and Kraut [14] find that targeted messages from strong ties is associated with positive improvements in well-being while viewing messages from friends broadcasted to all of their friends and receiving one-click feedbacks were not associated with any improvement in well-being. Kim and Shen [15] find that directed communication activities on social media platforms are positively associated with life satisfaction for older users.

Schemer et al. [16] and Johannes et al. [17] do not find any substantial relationship between social media use and subjective well-being. Liu et al. [18] conduct a meta-analysis of 124 studies and find that the association between social media use and well-being depends on the type of social media use. This association is positive for social interactions and negative for content consumption. Huang [19] also conducted a meta-analysis of 61 studies and documented a weak correlation between time spent on social media and negative indicators including depression and loneliness. For a comprehensive list of all studies studying social media use and well-being, see literature reviews by Appel et al. [20], Meier and Reinecke [21], Dienlin and Johannes [22], Masur [23] and Orben [24]. Cheng et al. [5] combine a survey of Facebook users with their Facebook activities and find that subjects reporting problematic use of Facebook were also going through a major life event such as a breakup. This shows that confounding variables could be a major concern in previous studies associating social media use and well-being.

Orben et al. [6] use a large-scale longitudinal dataset and conduct a specification curve analysis to rigorously analyze the relationship between adolescent social media use and well-being. Most of the analyses report tiny, trivial and insignificant results. Moreover, they provide evidence for reverse causality showing that social media use predicts well-being in the future and vice versa. Similarly, Sewall et al. [25] also find little to no evidence that changes in social media or smart phone use predict psychological distress.

Another major concern related to existing studies is the use of self-reported usage data. Survey respondents are typically asked to report the average time they spend on the internet, social media and digital devices. Several papers show that self-reported measures of technology use (including social media usage) are poorly correlated with actual usage and contain systematic patterns of misreporting [26-29].

Given this inconclusive evidence and lack of objective technology use data in existing literature, it is essential to obtain reliable causal evidence in a timely manner to inform policy makers. We aim to resolve this gap by obtaining evidence through a randomized controlled trial and using objective technology use metrics tracked by a software installed on the digital devices of our experimental subjects. In terms of outcome variables, we track measures of subjective well-being (life satisfaction and mental health) and performance (grades and number of credit points) over the duration of three quarters of an academic year (8 months) with the actual treatment lasting 2.5 months.

There is a related stream of literature using experiments to study the relationship between social media or computer usage and well-being or performance. Verduyn et al. [30] conduct a lab experiment where subjects are primed to passively use Facebook for 10 minutes and find that passive use is associated with a decline in subjective well-being. However, it is not straightforward if results from a 10-minute treatment can be generalized to long term effects.

Marotta and Acquisti [31] conduct an experiment with workers recruited from Amazon mechanical turk and offer productivity enhancing tools to subjects. One of the treatment groups has popular social media sites blocked during work hours. They find that workers in this group completed more tasks and increased their earnings. Carter et al. [32] conduct a randomized controlled trial in a US university where classes in the treatment group prohibited the use of computers in the class. They find that average exam scores were higher in the treatment group compared to the control group classes where students were allowed to use their computers. Using causal inference methods on observational data, Belo et al. [33] and Beland and Murphy [34] study the impact of broadband access and banning mobile phones in schools respectively on academic performance and also find evidence suggesting that digital distractions during class reduce academic performance. Taken together, evidence seems to suggest that digital device use in class or at work is harmful for student or worker performance. However, the overall causal impact of social media usage in life (inside and outside class or at work) on performance and well-being still remains an open question. Our study complements this research by analyzing the overall long-term impact of social media on well-being and performance as the subjects in our treatment group has restricted use of social media throughout their day for a long period of time.

A study closely related to our research is the experiment conducted by Allcott et al. [35]. They conducted a randomized controlled trial of Facebook users where subjects in the treatment group had to deactivate their Facebook account for 1 month. They find that this treatment reduced total online activity including other social media and this reduction persists after the end of the experiment. However, they use self-reported metrics of usage of online activities which are weakly correlated to objective usage metrics according to previous research. They measure 11 different metrics of subjective well-being and find that deactivating Facebook led to increase in subjective well-being for 4 out of the 11 metrics. Other related experiments include Brailovskaia et al. [36] (two weeks intervention, self-reported Facebook usage measure), Hall et al. [37] (one to four weeks intervention, self-reported Facebook usage measure), Hinsch and Sheldon [38] (one week intervention, self-reported Facebook usage measure), Tromholt [39] (one week intervention, self-reported Facebook usage measure), Mosquera et al. [40] (one week intervention), Hanley et al. [41] (one week intervention) and Vanman et al. [42] (five days intervention). Overall, the magnitude of the effects is small and it is not clear if these effects would have persisted for a treatment of longer duration. For a longer treatment duration, subjects could learn to live in a world without Facebook by discovering alternative substitutes providing similar use cases and their subjective well-being scores could go back to pre-experiment levels.

Methods

Experimental procedure

We recruited students in the faculty of economics and business of a large European university to take part in an academic study. The study received approval from the Institutional Review Board of the university. We used a flyer to invite students in lectures and from the pool of participants of the behavioral research lab of the university. The flyer informed students about the subject of the study, the required activities, the reward, and about measures to protect the participants’ privacy. Specifically, we let the students know that the study required to install a software (RescueTime) on their computers and mobile devices that keeps track of their digital activities and that allows them to analyze how much time they are spending on various categories of activities. We also stated that the study tracks their academic performance and well-being. Moreover, we informed the students that, in order to qualify for the reward, they need to keep the software running during the time of the study and to take part in four online surveys; one at the beginning of the study and one after each quarter. In addition to getting the software for free, we offered students €20 and a one out of 100 chance to win €1,000 if they take part until the end of the study.

The sign-up link forwarded interested students to a registration form that provided a more detailed privacy statement (about aim and principles of processing personal data, the types of data used, the limited recipients of the data, the storage period, and the students’ rights), informed consent, and asked students for their student email address, basic information about their studies (program, year), and the number and type of computers and mobile devices. The registered students were then invited to the study according to the experimental design detailed below.

Experimental design

The recruitment of students took place in the first quarter of the academic year 2018/19. We scheduled the experiment to run for the remaining three academic quarters. We will refer to these three terms as block 1, 2, and 3 of the study (which are quarter 2, 3, and 4 of the academic year). Each block consists of seven weeks of teaching and two examination weeks. The specific timeline was:

Block 1: from mid-November to end of January, with holidays from December 24 to January 3.

Block 2: from February to mid-April.

Block 3: from mid-April to end of June, with holidays from April 19 to 24.

We used the first block to establish a baseline of the students’ digital activities. In block 2, we randomly assigned participants to one of two conditions: a control group without specific instructions and a treatment group that received an incentive to use social media as little as possible. Specifically, we instructed them to use Facebook, Instagram and Snapchat for a maximum of 10 minutes per day. We did not block these services completely because not having access to social media at all might have a negative effect on students, e.g., if they use it to exchange important information about their studies. The 10-minute limit enables students to still access relevant information while not allowing them to waste a longer period of time. This is consistent with the Goldilocks hypothesis according to which moderate digital use may be advantageous compared to no use or overuse [43]. The software would inform students in the treatment group when they reached the limit and automatically block Facebook, Instagram and Snapchat afterwards. Students could disable this feature if they needed to use these services for longer. We informed students that we gave away another €1,000 among all students who achieved to stay under the 10-minute limit throughout block 2. Block 3 served to assess post-treatment effects.

We focus on Facebook, Instagram, and Snapchat in our experiment because they were the most popular social media services according to our measurement in block 1. As a caveat, the distinction of these services from others such as instant messaging can be debated. We return to this issue in the discussion.

We invited students to four surveys in total. We have sent the first survey in the first week of block 1. This survey asked students to give informed consent and, after referring them to the privacy statement, their agreement to use their academic grades for the purpose of the study. Moreover, we asked them about basic demographic information, their study program, and additional work activities next to their studies. Moreover, we provided measures of subjective well-being (see specific measures below). Upon completion of this first survey, we gave students the installation and registration instructions for the tracking software and asked them to keep this software running henceforth on all their computers and mobile devices. While the software was supported by Windows, OS X, and Android devices, it was not compatible with iOS devices (iPhone or iPad). In order to make sure that students with iOS devices complied to the 10 minute limit in the treatment condition, we informed them that we will ask them at a random time to hand in a screenshot of the Screen Time feature of iOS that reports similar information.

Surveys 2, 3, and 4 followed after each block and repeated the subjective well-being measures in order to track students’ well-being over time. We gave students a one-week deadline to fill out each survey.

Survey measures

As measures of subjective well-being we use the satisfaction with life scale (SWLS) [44] that consists of five items (In most ways my life is close to my ideal; The conditions of my life are excellent; I am satisfied with my life; So far I have gotten the important things I want in life; If I could live my life over, I would change almost nothing). These items are measured on a 7-point scale (1 “strongly disagree” to 7 “strongly agree”). SWLS is the most widely used scale to measure subjective well-being and is also used in previous studies studying social media use and well-being (e.g., [30,45]). In addition to SWLS, we also collected direct measures of happiness and life satisfaction through standard questions widely used in previous literature. Besides numerous other studies, the happiness question is used in the World values survey [46] and the life satisfaction question is used by Gallup [47] to calculate its well-being index. These questions are highly correlated with objective measures of well-being such as brain activity, emotional expressions and suicide rates as well as decision utility [48]. We obtain qualitatively similar results using these happiness and life satisfaction scores as we found using SWLS.

For measuring mental well-being, we adopted the shortened Warwick-Edinburgh Mental Well-being Scale (SWEMWBS) [49,50] with seven items (I’ve been feeling optimistic about the future; I’ve been feeling useful; I’ve been feeling relaxed; I’ve been dealing with problems well; I’ve been thinking clearly; I’ve been feeling close to other people; I’ve been able to make up my own mind about things). These items are assessed on a 5-point scale ranging from “None of the time” to “All of the time”. The SWEMWBS is a popular scale to measure mental well-being and is used in previous studies studying technology use and mental well-being [43].

Overview of data sources

Overall, our study makes use of three data sources: digital activities tracked by the software, self-reported measures via surveys, and academic grades from the educational administration. Table 1 shows an overview of these data types.

Table 1

Overview of data sources.

Type	Measure	Source	Data collection
Digital activities	Usage in number of seconds	Tracked by software	On each participant’s device throughout the entire study
Subjective well-being	Rating scales	Self-reported in surveys	At the beginning of the study and after each teaching block
Academic grades	From 0 to 10, with <6 = failed, 6 = below standard, 7 = standard, >8 above standard;	Educational administration	Once at the end of the academic year

The software tracks users’ activities on each device in 5-minute intervals and records how many seconds a user has actively used a specific program, app, or website in this interval, ranging from 1 to 300 seconds. Specifically, it records the user id, the name of the activity, the system name (Windows, Mac OS, or Android), and a timestamp. Since we are specifically interested in social media activities of the three most used social network services Facebook, Instagram and Snapchat, we used lookup tables to classify activities accordingly. For example, Facebook usage could appear in the activities as “facebook.com”, “fb.com”, “messenger.com”, “Facebook for Android”, “Facebook for Windows”, etc. We gathered this list of activities in block 1 and used each of these activities to count toward the 10-minute limit for the treatment group in block 2.

The European university at which this study took place uses a grading system that ranges from 0 to 10. Any grade below 6 represents a fail. A grade of 7 is most common and often referred to as “standard”, a 6 as “below standard“, and an 8 or higher as “above standard”. The grade for a lecture typically consists of a combined grade of the final exam and assignments that have to be completed during the course.

Sample

A total of 191 respondents completed the first survey. As is typical for longitudinal studies, some students dropped out over time such that 157 students completed survey 2, 144 survey 3, and 121 the final survey. The survey participation corresponds to the number of participants who reported digital activities using the software (see Table 2).

Table 2

Number of participants.

Part	Number
Completed survey 1	191
Used software in block 1 (calibration)	149
Completed survey 2	157
Used software in block 2 (treatment)	134
Completed survey 3	144
Used software in block 3 (post-treatment)	125
Completed survey 4	121
Took courses in block 1, 2, and 3	158

The following results will be based on the sample that recorded activities for at least 30 days in block 1 and 2 and completed surveys 1, 2, and 3. We will analyze the post-treatment data from block 3 and survey 4 separately. From the 134 students who recorded activities in block 1 and 2, we were able to match 122 from all data sources, i.e., twelve students did not answer (one of) the surveys or did not follow courses in at least one of the blocks.

Despite the dropouts, most importantly, there are no significant differences between the treatment and the control group, in terms of gender (Chi-squared test, p = 0.471), age (t-test, p = 0.961), mobile device operating system (Chi-squared test, p = 1.000), number of years studying at the university (t-test, p = 0.334) or whether students are working next to their studies in block 1 (Chi-squared test, p = 0.974) or block 2 (p = 0.594) (see Table 3 for details). There are also no significant differences between those who started the study and those who dropped out in terms of gender (Chi-squared test, p = 0.701), age (t-test, p = 0.113), mobile operation system (Chi-squared test, p = 0.975), of work status in block 1 (Chi-squared test, p = 0.109) or block 2 (p = 0.169). However, there is a significant difference between these samples regarding the study year (t-test, p = 0.027) such that those who dropped out are more likely to be Bachelor degree students than Master’s students. One potential explanation is that Bachelor degree students are more likely to quit their studies and not have any courses or grades registered subsequently.

Table 3

Descriptive statistics of the sample.

	Treatment	Control
Number of students	60	62
Gender: Female (vs. male)	0.467	0.548
Age (SD)	22.1 (3.3)	22.1 (3.1)
Mobile device operating system: Android (vs. iOS)	0.500	0.500
Studying in first to third year (vs. more than three years)	0.667	0.629
Working next to studying (block 1)	0.400	0.419
Working next to studying (block 2)	0.500	0.435

For the subsequent analysis, we note that at 80% power and alpha = 5%, the minimum detectable difference with our sample size is 2.5 for SWLS (assuming a standard deviation of 5.0), 1.5 for SWEMWBS (standard deviation = 3.0), and 0.5 for academic grades (standard deviation of 1.0).

Results

Digital activities

Social media usage

On average, students tracked 223.7 minutes of digital activities per day across the entire study (SD = 115.1 minutes). Students who use an Android mobile device recorded significantly more activities (265.6 minutes; t-test, p < 0.001), compared to students with an iOS device (182.5 minutes) as iOS was not supported by the software. While our activity estimates are more accurate for Android users we expect the treatment condition to be equally effective for both of these segments because we informed participants to also inspect their iOS tracked activities (see above).

Fig 1 shows the total number of minutes tracked by day, averaged for students with Android mobile devices in the treatment (black dots) and control groups (white dots). We report the activities for users with Android devices because the tracking is more accurate as it captures activities on desktop/laptop computers and their mobile devices (figures corresponding to the overall sample are in the Supporting Information, S1 Fig). The solid vertical lines separate blocks 1, 2, and 3 and the dashed vertical lines indicate the start of the examination period. Overall, digital activities remain on a high level each day but are reduced during the winter holiday season and during the examination periods.

Fig 1

All digital activities and social media usage over time (users with Android devices).

As a manipulation check, the bottom part of Fig 1 shows activities for social networking (Facebook, Instagram, and Snapchat combined) for the Android sample. The mean daily usage in minutes is 21.1 minutes (27.9 minutes) for users in the treatment (control) group in block 1, which is not significantly different (t-test, p = 0.310). The incentive to reduce social media activities was effective as students in the treatment condition significantly reduced their social media usage in block 2 compared to the control group (t-test, p = 0.009). The horizontal line represents the 10-minute limit imposed on the treatment group. The average usage per day is close to the limit in the treatment group with 8.1 minutes. Within the control group, the average daily usage of 24.2 minutes in block 2 is on the same level as in the first block (paired-sample t-test, p = 0.245).

Remarkably, although students in the treatment group significantly reduced their social media activities, their overall digital activities overall are not affected but, in fact, exceed those of the control group in block 2 (t-test, p = 0.026). This result indicates that students substituted or even overcompensated their social media usage with other activities.

Substitution. Fig 2A and 2B show the time series of activities of users with an Android device for the most used categories of services (we exclude the categories of general utilities, which holds mostly operating system activities, and uncategorized services for which there are no significant differences between the groups; activities for all users are in the Supporting Information, S2A and S2B Fig). We find significant substitution for social networking with instant messaging (p = 0.008). Accordingly, more students in the treatment condition used instant messaging in block 2 when social media was restricted compared to block 1 and to the control group (difference-in-differences). The activities increased from an average daily use of 25.1 minutes in block 1 to 28.8 minutes in block 2 in the treatment group, while the usage decreased from 21.8 minutes to 15.2 minutes in the control group. Most activities (92.9%) in this category are related to WhatsApp.

Fig 2

a. Tracked digital activities over time (users with Android devices). b. Tracked digital activities over time (users with Android devices).

We also find a significant increase in usage of music for Android users (t-test, p = 0.027) in the treatment group in block 2. However, average daily activities in this category are rather low (below five minutes) and the difference is mostly driven by two outliers who listen to music for more than 30 minutes each day on average. Other activities show plausible patterns, e.g., the reference and learning category (activities include the university intranet, PDF reader, Wikipedia, Mendeley, Google scholar, EBSCO, etc.) shows peaks before the exam period. However, these and other activities are not affected by reduced social media usage (an overview of significance tests comparing treatment and control groups in block 2 vs. block 1 is given in the Supporting Information, S1 Table).

Subjective well-being

For the subjective well-being measures, the SWLS and SWEMWBS items show high reliability (Cronbach’s α being 0.84 and 0.76 respectively). For the analysis we calculate scores as the sum of their items (with SWEMWBS being transformed according to a defined conversion table). The students score averages (SD) in SWLS of 25.0 (5.5), 25.0, (5.4), 25.1 (5.3) in the three surveys at the beginning of the study and after block 1 and 2. That means they are between an “average” and “high” score of satisfaction. Treatment and control group are not significantly different at the beginning of block 1 (t-test, p = 0.182; survey 1), at the end of block 1 (t-test, p = 0.212; survey 2), or, most importantly, at the end of block 2 after the exposure to the treatment (t-test, p = 0.167; survey 3). The same implications hold for the SWEMWBS scores that shows average scores (SD) of 22.8 (3.0), 22.5 (2.7), and 22.4 (3.3) in the three surveys (see S2 Table in the Supporting Information for details).

Fig 3 plots the differences between survey 3 and survey 2 (before and after the social media restriction) in terms of SWLS and SWEMBS. The distributions are centered on zero, illustrating the non-significant difference between the treatment and control group.

Fig 3

Differences in subjective well-being measures.

Table 4 shows correlations with the subjective well-being measures and the digital activities in block 1 (i.e., activities that are not affected by the treatment condition) for users with an Android device. We detail the correlations of the subjective well-being measures at the beginning of the first block (in survey 1) and at the end of the block (in survey 2) to study potential reverse causality, e.g., increased well-being leads so more/less social media activities or vice versa.

Table 4

Correlations of subjective well-being measures and digital activities in block 1 (Android users).

	SWLS Survey 1 (start of block 1)	SWLS Survey 2 (end of block 1)	SWEMWBS Survey 1 (start of block 1)	SWEMWBS Survey 2 (end of block 1)
SWLS Survey 2 (end of block 1)	0.80
SWEMWBS Survey 1 (beginning of block 1)	0.76	0.63
SWEMWBS Survey 2 (end of block 1)	0.63	0.72	0.77
All digital activities	-0.06	-0.11	-0.15	-0.05
Social media usage	-0.03	-0.02	-0.13	-0.02
Social media usage low	-0.31	-0.23	-0.08	-0.13
Social media usage medium	0.26	0.19	0.14	0.15
Social media usage high	-0.02	-0.01	-0.08	-0.05
General Reference & Learning	-0.13	-0.14	-0.20	-0.06
Instant Message	0.18	0.24	0.14	0.27
Browsers	-0.09	-0.13	-0.01	-0.11
Video	-0.28	-0.27	-0.28	-0.21
Writing	0.24	0.27	0.10	0.09
Search	-0.19	-0.19	-0.22	-0.10
Email	0.21	0.28	0.11	0.10
General News & Opinion	0.19	0.04	0.14	0.13
Games	0.06	-0.17	0.04	-0.04
Presentation	0.14	0.10	-0.01	0.03
General Shopping	-0.05	0.02	-0.12	0.02
Music	0.12	0.10	0.12	0.24

(Correlations in bold font are significant on a 5% level).

Satisfaction with life and mental well-being are positively correlated and are also significant predictors over time, i.e., subjective well-being in survey 1 is positively correlated with the same measure in survey 2. Regarding digital activities, we see, on average, negative correlations between subjective well-being and all digital activities, albeit not being significant. Similarly, we do not find significant correlations with social media use. (We will address causality in the section below.) To address potential non-linear effects we also report correlations with categories of social media usage. Specifically, we used dummy variables relating to low usage with an average of less than 2 minutes per day (36.1% of users), medium usage of 2 to 20 minutes (39.5%), and high usage of 20 minutes or more (24.4%). A non-linear relationship is likely as low usage generally shows the most negative subjective well-being scores, while medium usage and not high usage scores the highest well-being. However, we cannot rule out reverse causality regarding these findings as the only significant relationships are between satisfaction with life measured in survey 1 and the social media activities measured after the survey has taken place.

Activities related to communication, i.e., instant messaging and email, show significant positive correlations in the second surveys (for instant messaging regarding mental well-being and for email in terms of satisfaction with life). A consistent significant negative correlation can be observed for activities in the video category and satisfaction with life (both surveys) and mental well-being (survey 1). This suggests that less satisfied students and those with lower mental well-being at the beginning of the block increasingly watch videos in the subsequent block.

Academic performance

The students participating in our study scored an average (SD) of 7.105 (1.078) in block 1 and 7.122 (0.954) in block 2. Most grades can be classified as “standard”. The average (SD) sum of credit points per block is 13.571 (5.754) in block 1 and 12.353 (5.118) in block 2. Differences between the treatment and control group are not significant for grades (t-test, p = 0.113) but for the number of credit points such that the treatment group attempted to score significantly more credits (t-test, p = 0.035). This is visualized in Fig 4 as the difference in grades and credits in block 1 and 2. Note that the number of ECTS represents the courses that the student attempted to pass but they are also stored if the student failed the exam. A comparison of the number of successfully passed courses shows no significant differences between the groups (t-test, p = 0.383).

Fig 4

Differences in academic performance.

Table 5 shows the correlations of academic performance in block 1 with the subjective well-being measures and digital activities. Accordingly, grades in block 1 are positively related to grades in block 2. The grades are not significantly correlated with the number of credit points, possibly due to a trade-off regarding a good grade and completing more courses. Credits are also not positively related over time, which is reasonable as more credits in one term means that the students have to obtain fewer credits in subsequent terms. The average grade is positively and significantly correlated with satisfaction with life measures. This holds for SWLS measures at the beginning and end of the block.

Table 5

Correlations of academic performance with measures of subjective well-being and digital activities in block 1 (Android users).

	Average grade in block 1	Sum of credit points in block 1
Sum of credit points in block 1	-0.16
Average grade in block 2	0.46	-0.19
Sum of credit points in block 2	0.06	0.01
SWLS Survey 1 (beginning of block 1)	0.27	-0.10
SWLS Survey 2 (end of block 1)	0.31	0.02
SWEMWBS Survey 1 (beginning of block 1)	0.16	-0.07
SWEMWBS Survey 2 (end of block 1)	0.22	0.01
All digital activities	0.16	-0.16
Social media usage	0.02	-0.03
Social media usage low	-0.08	0.21
Social media usage medium	0.06	-0.26
Social media usage high	0.00	0.09
General Reference & Learning	0.13	-0.21
Instant Message	0.28	-0.28
Browsers	-0.03	-0.18
Video	-0.04	0.15
Writing	0.25	0.05
Search	-0.13	-0.21
Email	0.12	-0.21
General News & Opinion	-0.07	-0.10
Games	-0.06	-0.08
Presentation	0.28	0.17
General Shopping	-0.02	-0.01
Music	-0.08	-0.13

(Correlations in bold font are significant on a 5% level).

Regarding correlations with digital activities we can observe significant positive effects on the average grade for writing and presentation activities, which are required to complete assignments (that are part of the grade for the majority of courses). We also see a positive effect of instant messaging on the grade, however, a negative effect on the number of credit points. Social media usage is not significantly correlated with the academic performance in block 1. Only when categorizing students based on their social media usage we see significant effects such that medium usage is negatively correlated with the number of credits.

To what extent these findings can be interpreted as causal evidence will be addressed in the following section by analyzing the complete randomized control trial across the two blocks using difference-in-differences analyses with regression models.

Regressions

Tables 6 and 7 show the results of the regression models that use subjective well-being (Table 6) and academic performance measures (Table 7) as the dependent variables. We use data from the two teaching blocks with “block 2” being a dummy variable indicating the block in which the treatment took place. Similarly, the “treatment group” refers to a dummy variable that identifies students that were exposed to the treatment. We use gender, age, number of years at the university, and whether the student is working next to the studies (dummy variable) as control variables.

Table 6

Regression of satisfaction with life and mental well-being measures (standardized coefficients in parentheses).

	Satisfaction with life (SWLS)				Mental well-being (SWEMWBS)
	All users		Android		All users		Android
	Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value
(Intercept)	29.028	<0.001	36.967	<0.001	24.409	<0.001	30.497	<0.001
Treatment group	-1.205(-0.113)	0.221	-1.350(-0.117)	0.380	-0.233(-0.038)	0.676	-0.063(-0.009)	0.942
Block 2	0.207(0.019)	0.831	-0.552(-0.048)	0.720	0.025(0.004)	0.963	-0.342(-0.050)	0.691
(Treatment group*Block2)	-0.253(-0.020)	0.855	1.285(0.097)	0.552	-0.126(-0.018)	0.872	0.505(0.065)	0.676
Gender (female)	-1.021(-0.096)	0.143	-0.818(-0.070)	0.455	-0.991 (-0.162)	0.013	-0.768(-0.112)	0.210
Age in years	-0.161(-0.096)	0.221	-0.494(-0.217)	0.068	-0.057(-0.059)	0.446	-0.338 (-0.252)	0.026
Years at the university	0.166(0.048)	0.542	0.107(0.028)	0.798	-0.139(-0.069)	0.370	-0.089(-0.040)	0.705
Working next to studies	0.280(0.026)	0.692	-0.036(-0.003)	0.975	0.724(0.117)	0.072	1.020(0.150)	0.117
R-squared	0.032		0.051		0.055		0.142

Table 7

Regression of academic performance (standardized coefficients in parentheses).

	Regression of average grade				Regression of number of credit points				Regression of number of courses passed
	All users		Android users		All users		Android users		All users		Android users
	Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value
(Intercept)	6.950	<0.001	6.578	<0.001	19.406	<0.001	13.393	0.009	3.904	<0.001	2.616	0.007
Treatment group	0.237(0.117)	0.205	0.536(0.244)	0.067	-1.740(-0.159)	0.071	-1.159(-0.108)	0.388	-0.064(-0.032)	0.721	0.124(0.064)	0.625
Block 2	0.173(0.086)	0.350	0.321(0.146)	0.272	-2.750 (-0.252)	0.004	-2.931 (-0.273)	0.031	-0.347(-0.172)	0.051	-0.345(-0.177)	0.178
(Treatment group*Block2)	-0.310(-0.132)	0.239	-0.592(-0.235)	0.149	3.090 (0.245)	0.023	3.431(0.278)	0.070	0.220(0.094)	0.382	0.211(0.094)	0.554
Gender (female)	0.142(0.070)	0.281	0.137(0.062)	0.507	0.706(0.065)	0.300	0.388(0.036)	0.684	0.201(0.099)	0.113	0.090(0.046)	0.618
Age in years	-0.016(-0.048)	0.534	0.002(0.005)	0.969	-0.130(-0.076)	0.311	0.164(0.078)	0.483	-0.055 (-0.172)	0.022	0.003(0.007)	0.949
Years at the university	0.114(0.172)	0.028	0.083(0.116)	0.294	-0.854 (-0.239)	0.002	-1.232 (-0.352)	0.001	-0.109 (-0.165)	0.028	-0.131(-0.206)	0.060
Working next to studies	-0.043(-0.021)	0.750	-0.202(-0.092)	0.358	-0.009(-0.001)	0.990	1.439(0.134)	0.156	-0.082(-0.040)	0.523	0.134(0.069)	0.484
R-squared	0.034		0.054		0.117		0.159		0.113		0.083

Regarding subjective well-being measures SWLS and SWEMWBS, we do not see any significant effects due to using Facebook, Instagram, and Snapchat less. The (Treatment group * Block2 interaction) that indicates if the treatment group differed from the control group in block 2 remains insignificant, irrespectively of the well-being measure (SWLS or SWEMWBS) or sample used (full sample or Android subset). Only few predictors are significant (gender in the full sample, p = 0.013 or age in the Android subset, p = 0.026; both for the SWEMWBS) but these are not directly related to the experimental setting. Overall, the amount of variance explained remains rather low and ranges between 3.2% (SWLS measure, full sample) and 14.2% (SWEMWBS, Android sample).

There is also no evidence that the treatment group achieves higher grades in block 2 (Treatment group * Block2 interaction), in which social media usage was restricted, than the control group (p = 0.239). This also holds for other subsets of the sample such as students with an Android device. Overall, the amount of variance explained in the grades remains very low with 3.4%. Only the number of years that the student has spent at the university is a significant positive predictor, i.e., Master’s students achieve higher grades than Bachelor’s.

In terms of number of credit points, i.e., number of courses attended, we do see a significant effect of restricting social media. While students overall attended courses with fewer credits in block 2 (p = 0.004) this is not the case for students in the treatment group such that they targeted significantly more credits (p = 0.023). With this dependent variable, the number of years at the university has a significant negative effect (p = 0.002) and, overall, 11.7% of variance in the number of credit points can be explained. The subset of Android users replicates these results, albeit generally with lower levels of significance due to the smaller sample size.

However, as noted above, the number of credit points does not necessarily show that students successfully passed more courses as also failed courses are included. Using the number of courses passed as the dependent variable shows that the students in the treatment group in fact do not differ from the control group (a Poisson model replicates these results). Thus, it appears that the treatment group attempted to pass more courses or courses with more credits compared to the control group but did not necessarily succeed.

These null results of the effects of restricting social media usage on academic performance and subjective well-being raises the question of whether our study is underpowered to detect economically significant effects. At 80% power the minimum detectable difference in life satisfaction (on the SWLS scale) with our sample size is 2.5 on a scale of 5–35 (average life satisfaction score in our sample is 25 with a standard deviation of 5). The SWLS categories comprise 5 scale points, for example, a score of 25–29 is considered as a “high” score. Therefore, even if the treatment group’s life satisfaction score increased by 1.5 it is generally not sufficient to change the classification from one category to another. The minimum detectable difference in average grade is 0.5 on a scale of 1–10 (average grade in our sample is about 7 with a standard deviation of 1). Given that students receive grades which are whole numbers, this threshold is still below the value that would increase the treatment group’s average grade by a full point. At least, our results indicate that changes in grades and subjective well-being are not substantial so that they could be detected with our sample size.

To further address potential concerns about statistical power we applied a hierarchical Bayes ANOVA (BANOVA) model that includes between and within subject effects and accommodates unobserved heterogeneity by including a normal distribution of the parameters across individuals [51]. All models converge and generally replicate the results above (details are available from the authors upon request).

Post-treatment effects

The formal analysis of the post-treatment effects is based on a sample of 106 students who provided activity data throughout all three blocks and in all four surveys. While the treatment condition significantly reduced their social media usage in block 2 compared to the control group, this effect was not permanent. After we suspended the limit in block 3 the social media activities of users in the treatment group increased again, showing no significant differences to the control group any longer (t-test, p = 0.668). We further do not see any significant differences between the treatment and control group in block 3 in terms of grades (t-test, p = 0.152), number of credit points (t-test, p = 0.923), satisfaction with life (t-test, p = 0.499), or mental well-being (t-test, p = 0.966), i.e., there is no lagged effect of reduced social media usage.

Discussion

In this paper, we analyzed the effects of restricting social media usage. We did not find significant causal effects of social media usage on well-being or academic performance, other than students attempting (but not succeeding) to pass more courses or courses with more credits. However, we found robust evidence of substitution effects that can potentially explain the null finding. Specifically, we showed that social media and instant messaging apps can be substitutes. The European Commission approved Facebook’s acquisition of WhatsApp in 2014 based on Facebook’s claim that it operates in a different market and does not compete directly with WhatsApp [52]. Our results indicate that they are in fact direct competitors. After acquiring WhatsApp, Facebook started automatically matching its users’ profiles with their WhatsApp accounts and has started integrating WhatsApp, Instagram and Facebook user accounts [53]. The European Commission fined Facebook €110 million in 2017 for this practice because Facebook had provided misleading information about the feasibility of automatically matching profiles during its acquisition of WhatsApp. However while announcing this fine, the Commission still maintained its belief that Facebook and WhatsApp do not directly compete with each other [54]. Antitrust authorities should consider the market power of this combined entity if the world’s biggest social media platforms are integrated with the world’s biggest instant messaging platform. This finding also raises the question how to properly define social networks, for future academic studies or antitrust cases.

While we found null results estimating the causal impact of social media usage on well-being and academic performance, and not all null results matter, we believe that null results are interesting and important in this context. The media has hyped correlational studies showing a negative association between social media usage and well-being and it is important to balance this narrative through causal evidence. A limitation of our study is the lack of a larger sample size to detect smaller effects. While these small effects might not be economically significant, more research is needed using massive samples. Future research can also look at differences between Android and iOS users in more detail with larger samples since we lack power to analyze these differences in our current study. However, it is challenging to recruit a large number of subjects from a representative sample for a long-term study. Direct collaborations with social media platforms or internet service providers (which control internet traffic) could be a way of obtaining data from larger samples. These would also facilitate more targeted interventions such as restricting only content consumption or social interactions on social media [18].

It is interesting to notice that while social media generates large amount of consumer surplus [3], it doesn’t seem to affect the subjective well-being of users. Future research can explore this wedge between consumer surplus and subjective well-being and see whether they are correlated for some products and uncorrelated for others. Future research should also explore the addictiveness of social media in more detail [55]. Our findings in block 3 show that the students in the treatment condition go back to their old habits and do not adopt a lower social media usage that they experienced in block 2. On the other hand, showing students how much time they are spending on social networks via the software seems to have an overall negative trend on its usage (comparing usage in block 1 and block 3). Curing social media addiction (if it is indeed addiction) might therefore be a longer process. Future research can look at mechanisms for the emergence of social media addiction, for e.g. through targeted advertising or news feed algorithms and features of social media apps (e.g. video sharing) which are correlated with addiction.

Moreover, due to our student sample implications for the general population are limited. It could be that students use social media mostly for communication purposes and therefore show significant substitution effects with instant messaging. We might see different effects for users who visit social media for content consumption, e.g., watching videos, instead of social interaction [18]. In this regard, newer social media services such as TikTok might have a different effect. However, these and other comparable social media services were not relevant at the time we conducted the study (as we measured in block 1). More research is needed looking at emerging social media platforms such as TikTok. Moreover, one could define social media more broadly to include instant messaging services and implement a stricter restriction involving all types of communication and social networking apps. However, conducting such a study for any length of time is a challenging endeavor.

Additionally, while we study the impact of social media on students and academic performance, future research can look at workplace settings and study the impact of social media and its substitutes on worker productivity and well-being. We believe that rigorous causal evidence through randomized controlled trials and objectively measured time spent is the way forward in addressing questions regarding the impact of technology on well-being.

The widespread adoption of most major technologies in the past such as radio, television, video games and computers was followed with unfounded fears about their impact on well-being. This story repeats again with social media. We find that social media usage does not cause lower well-being or poor academic performance. Rather, we demonstrate that students find other means of social networking using instant messaging when exogenously restricting their social media usage. To conclude: You can take social networking away from the students, but you cannot take students away from their social network.

All digital activities and social media usage over time (all users).

(DOCX)

Click here for additional data file.

a Tracked digital activities over time (all users). b. Tracked digital activities over time (all users).

(DOCX)

Click here for additional data file.

Significance of differences between treatment and control group in block 2 vs. block 1 (difference-in-differences).

(DOCX)

Click here for additional data file.

Summary statistics for well-being measures.

(DOCX)

Click here for additional data file.

20 Aug 2021

PONE-D-21-20386

Effects of restricting social media usage

PLOS ONE

Dear Dr. Collis,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

My summary of the major concerns raised by the reviewers is below and will hopefully give you guidance in how to respond if you feel that you can.

Please submit your revised manuscript by Sep 24 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see:  http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at  https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Daniel Romer

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please consider changing the title so as to meet our title format requirement (https://journals.plos.org/plosone/s/submission-guidelines). In particular, the title should be "Specific, descriptive, concise, and comprehensible to readers outside the field" and in this case it is not informative and specific about your study's scope and methodology.

3. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

4. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please delete it from any other section.

Additional Editor Comments

We have received two reviews that I think will be helpful to you if you wish to pursue publication. In particular, both reviewers think your study is under-powered considering the typical effect size that is observed in the literature. This may be a serious obstacle to moving ahead with the study as it stands. However, R2 suggests that perhaps a multi-level analysis may increase your power and thus overcome this problem,. R1 also suggests that it may be valuable to examine effects in terms of mediators that have produced different effects and thus hindered your ability to observe an effect (i.e., the type of content that is accessed). If this objection can be overcome, then it may be possible to salvage your study. In addition, both reviewers think that your study is limited in so far as it only required reductions in use of three platforms. Can you respond to this concern by looking at what the students actually did in each condition? Obviously, if the experimental group merely displaced use to other platforms, then the findings are much more limited in terms of overall social media use. In addition, from my own perspective, I would look to see if any effects are more likely among those who were already experiencing mental health problems because withdrawal from in-person interaction is common among those with depression and related conditions. it may be that they are the ones who are likely to show stronger effects of not using the one socializing outlet they find attractive. Finally, R1 asks for more disclosure of findings that would be helpful to others who would want to include your findings in a synthesis of results, and both reviewers feel that important prior papers are being ignored in your literature review.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Review for PONE-D-21-20386 : Effects of restricting social media usage

This paper describes a random assignment experiment comparing the subjective well-being (satisfaction with life and mood) and grades of university students who were asked to reduce their social media use (Facebook, Instagram, Snapchat) to less than 10 minutes per day with a control group who were not asked to make social media adjustments. Results show no significant differences between the reduction group and control group on these outcomes. (Because I am not familiar with work that on social media use and educational outcomes, I will ignore it in this review.)

As the authors note, prior research on the impact of social media use on subjective well-being has primarily involved cross-sectional surveys, correlating self-reports of social media use with self-reports of well-being. Moreover, again as the authors note, results from the correlational research have been mixed, although the most recent meta-analysis reviewing 94 studies (Liu et al., 2019) suggests that on average social media use is negatively associated with subjective well-being. However, the association is substantially dependent on type of use (e.g., r=.14 for social interaction versus -.14 for content consumption). Thus, a random assignment experiment is an excellent addition to this body of work.

The authors introduce their research with the claim for novelty. “To the best of our knowledge, this study is the first that tracks all of these components of well-being and over a long period of time.” This claim is overstated. I am familiar with five experiments, in which participants were randomly assigned to reduce their social media use for one to four weeks (Allcott et al., 2020; Brailovskaia et al., 2020; Hanley et al., 2019; Mosquera et al., 2019; Vanman et al., 2018). The current research improves on prior experiments by upping the ante to nine weeks. Although some of the prior experiments used small student samples like the current study, others used much larger and more diverse samples. All of them show that social media restrictions led to better subjective well-being. Of these, the current authors cite only the Allcott paper, but criticize it because it only reduced social media use for a four-week period.

I have some substantive methodological concerns about the paper involving the conceptualization of the intervention, statistical methods, and the power of the intervention. Given the prior literature, which suggests that the impact of social media use depends upon the functions for which it is used and not just the total amount of use, I am disappointed that the current research did not target more precisely the type of social media use to restrict. For example, a 2 X 2 factorial experiment (reduce social interaction vs control (reduce content consumption vs control) would have been more informative than the current experiment, which restricted overall use, and could have been done even with a relatively small sample size of 130 completed participants.

I also disappointed on the way that the research defined social media use for participants. The research considered Facebook messenger (a popular application on Facebook), Instagram and Snapchat, all of which are used for one-on-one or small group communication, as social media applications while WhatsApp, another messaging application, is not classified as social media.

According to the methods description on page 13 and Tables 4 and 5 in the manuscript, the authors have some data on what activities participants used across the platforms. I think the authors could more strongly advance the prior literature about the differential importance of types of use and still take advantage of their random assignment experiment by using instrumental variable regression or mediation models to analyze the data. For example, the authors could use the experiment to dig more deeply into the conclusion form Liu et al’s meta-analysis that social interaction and content consumption have different effects on well-being. An instrumental variable regression analysis could examine the extent to which the experimental manipulation changed participants’ type of use (e.g., instant messaging and email as proxies for use for social interaction vs general news, general shopping, and video as proxies for content consumption) and how the changes in in use for social interaction and content consumption use caused by the manipulation predict well-being and grades.

I have two concerns about the power of the experiment. First, the manipulation check indicates that the experimental manipulation did not have a major impact on the amount of time participants spent on social media, with participants in the experimental condition reducing time spent on the three predefined social media applications by 13 minutes versus 3.7 minutes for participants in the control condition. Given the substitution behavior the authors document, in which participants in experimental condition started using WhatsApp more, the experimental manipulation did not reduce overall social media use. Instead, it caused shifts in the types of social media applications used, if one considers Facebook, Instagram, Snapchat, and WhatsApp all as social media platforms. This shift in behavior is another reason I recommend that the authors consider an instrumental variable regression model to test their hypotheses.

Second, both the correlational and experimental prior literature suggests that the relationship between social media use and well-being is likely to be small, with the average absolute correlation being -.10 in the Liu meta-analysis of correlational studies and a .09 standard deviation (STD) effect size in the Allcott et al experiment. Given the likely small effect size, an experiment with 130 participants is probably underpowered to detect this effect. The authors’ discussion of their power analysis indicates that their study could detect a .6 STD effect on the SWLS Life Satisfaction scale, substantially larger than the effect size one might have anticipated from the prior literature. While the authors acknowledge this limitation, they still interpret their failure to find a significant effect of their restriction intervention much more seriously than it deserves. As their paper states, “Contrary to findings from previous correlational studies, we do not find any impact of social media usage on wellbeing and academic success. … While we found null results estimating the causal impact of social media usage on well-being and academic performance, and not all null results matter, we believe that null results are interesting and important in this context. The media has hyped correlational studies showing a negative association between social media usage and well-being and it is important to balance this narrative through causal evidence.” Indeed, even though the experiment found no significant effects of their intervention on either satisfaction with life or mental well-being, the treatment X Block2 effects are both negative and consistent with the prior literature. As part of this review, I had intended to conduct a quick and dirty meta-analysis of the experimental literature to date (i.e., the current paper plus the five additional experiments listed below) but discovered that the authors of the current paper don’t provide enough information to include their research in the meta-analysis. Page 17, where the authors present their well-being results, should include a table of means and standard deviations for the SWLS and SWEMWBS scores for Block 2 broken down by experimental condition. Table A-3, which only reports the regression coefficients and p-values does not provide sufficient information to reconstruct the needed information.

In summary, this is an interesting study with a laudable goal. Experimental evidence is hugely important, and the current study supplements other recent experiments restricting social media usage by throttling back usage over a 9-week period, compared to a maximum of four weeks in other work. However, the study has some important flaws that make me believe the paper shouldn’t be published in its current form. The authors made arbitrary decisions about what online behavior to consider use of social media (i.e., only Facebook, Snapchat, and Instagram, but not WhatsApp), failed to differentiate types of use (e.g., social interaction versus content consumption) even though the prior literature suggest that type of use is an important moderator, and was under powered. A more minor problem is that the authors don’t provide enough information to allow their research to be used in a meta-analysis. I think the paper still has promise but needs to do more sophisticated data analysis (e.g., Instrumental variable approach mentioned previous or a mediation analysis) to link the changes induced by the experimental manipulation on well-being.

References

Allcott, H., Braghieri, L., Eichmeyer, S., & Gentzkow, M. (2020). The welfare effects of social media. American Economic Review, 110(3), 629-676.

Brailovskaia, J., Ströse, F., Schillack, H., & Margraf, J. (2020). Less Facebook use–More well-being and a healthier lifestyle? An experimental intervention study. Computers in Human Behavior, 108, 106332.

Hanley, S. M., Watt, S. E., & Coventry, W. (2019). Taking a break: The effect of taking a vacation from Facebook and Instagram on subjective well-being. PloS one, 14(6), e0217743.

Liu, D., Baumeister, R. F., Yang, C.-c., & Hu, B. (2019). Digital communication media use and psychological well-being: A meta-analysis. Journal of Computer-Mediated Communication, 24(5), 259-273.

Mosquera, R., Odunowo, M., McNamara, T., Guo, X., & Petrie, R. (2019). The economic effects of Facebook. Experimental Economics, 1-28.

Vanman, E. J., Baker, R., & Tobin, S. J. (2018). The burden of online friends: The effects of giving up Facebook on stress and well-being. The Journal of social psychology, 158(4), 496-508.

Reviewer #2: The paper submitted to PLOS One investigates whether voluntary restriction of social media usage to 10 minutes per day has any effects on a) subjective well-being and b) academic performance. To this end, the authors conducted an randomised controlled trial in which half of 121 students were asked to restrict their social media usage (i.e., the use of social network sites Facebook, Instagram and Snapchat) to a maximum of 10 minutes per day over a period of 1.5 months. Overall, the authors did not find any effects of social media usage restriction on subjective well-being or academic performance.

First, I would like to highlight that I deem the topic – social media uses effects on well-being/academic performance – and the proposed experimental approach to study it – experimentally restricting social media use – highly important and innovative. The debate around effects of social media use or screen time on well-being is on-going, both in public and scientific circles. And the authors are right to point out that much of it is based on questionable, correlational evidence. From this point of view, I welcome the submission. Second, the experiment is well designed and conducted, the analyses seem for the most part sound, and the paper overall well-written. The author neither make to strong claims, nor fail to articulate the contribution. As such, I do believe that the paper has the potential to add to the literature.

However, I do see several problems and issues, most minor but some also major, that require revision.

- First and foremost, I am concerned that the authors did not conduct comprehensive a priori power analyses. The sample sizes after each block are comparatively small and hardly sufficient to test effects of reduced social media use on general and stable outcomes such as well-being or academic performance. In other words, the study seems heavily underpowered to study the effects of interest. I believe that this is a major issues that unfortunately might not be resolvable. I thus recommend the authors to consider conducting another study with a more appropriate sample size. When doing so, I would also recommend to preregister all assumptions and analysis plans.

- Second, although the authors provide a summary of related literature, it – from my point of view – does not reflect the current state of the debate sufficiently. A lot of major contributions in the past years are missing and some similar experiments are not mentioned (among others, I was particularly missing reference to the work of Jeffrey Hall, 2021).

- Another issue relates to a lack of transparency in reporting. Some important aspects of the data analysis are not reported (e.g., missing data analysis and treatment, exact tests and covariates...). Information about the used tracking app (including how well the capturing of data worked) is missing completely. I appreciate that the authors explain why the data cannot be shared, but I strongly recommend to make everything else (material, instructions, information about the app, analysis code, etc.) available to the reader.

In the following, I will highlight more specific issues by going through the manuscript chronologically.

pp. 3-9.: As mentioned above, I am missing a comprehensive discussion of the current state of the literature. With regard the relationship between social media use and well-being, consider referring to the meta-analyses Huang (2010, 2017), Liu, Baumeister, Yang & Hu (2019), as well as the reviews by Appel, Marker & Gnambs (2020), Meier & Reinecke (2021), Dienlin & Johannes (2020), Masur (2021) and Orben (2020). Consider also looking at more recent studies on the relationship, e.g., Schemer et al. (2020), Kim & Chen (2020), Johannes et al. (2020). With regard to experimental studies on effect of social media abstinence, I was further missing reference to the work by Jeffrey Hall (2021) and the earlier works by Hinsch & Sheldon (2013) and Tromholt (2016). I strongly recommend that the authors revise this sections considerably.

p. 9: Consider explicitly formulating your hypotheses. In its current form, I find it difficult to understand what you are exactly testing.

p. 10: What was the exact reason to assess participants at 3 different time points? And why did you not incorporate this in your analysis? Wouldn't it be possible to run some sort of time-series or multi-level analysis on the entire data set? This way, you could potentially also increase your power (by combining within- and between-perspectives).

p. 11: The paper is framed as "reducing social media use". The term social media is quite broad. Often, it even includes instant messaging and other types of platforms. In light of this, why only restricting Facebook, Instagram and Snapchat use? What about TikTok, Twitter, Pinterest, Youtube? I suggest to include clear definition of social media and justify your choice of platforms. This may have important implications for how to frame the entire study and how to intrepret the results as well: I would be careful to generalize on all social media use given that the treatment was restricted to three platforms (albeit the most used ones).

p. 12: Although perhaps not very common, I would appreciate you reporting comprehensive factor analyses (CFAs, etc) for both well-being scales. Despite them being used a lot, they often show considerable misfit with the data.

p. 13: What type of software/application was used? Please provide more information and validity checks. Did the tracking work perfectly? From my experience, one often has some problems in the data with such software.

p. 14: What was your justification for the sample size? As mentioned earlier, I am concerned that your study is heavily underpowered. Further, how did you deal with missing data? Did all 122 participants answers all items in all waves?

p. 15: In reporting the results, it is often unclear what tests of analytical approaches were chosen. Please indicate more clearly which results relate to which analysis. Are the p-values related to simple t-tests? Or more elaborate ANOVAS including covariates? This should be made more clearer. Please report unstandarized and standardized coefficients (effect sizes) as well. Significance alone is not meaningful.

References

Appel, M., Marker, C., & Gnambs, T. (2020). Are social media ruining our lives? A review of meta- analytic evidence. Review of General Psychology, 24(1), 60–74.

Dienlin, T., & Johannes, N. (2020). The impact of digital technology use on adolescent well- being. Dialogues in Clinical Neuroscience, 22(2), 135–142.

Hall, J. A., Xing, C., Ross, E. M. & Johnson, R. M. (2021) Experimentally manipulating social media abstinence: results of a four-week diary study.Media Psychology, 24:2, 259-275, DOI: 10.1080/15213269.2019.1688171

Hinsch, C., & Sheldon, K. M. (2013). The impact of frequent social internet consumption: Increased procrastination and lower life satisfaction. Journal of Consumer Behaviour, 12, 496–505. doi:10.1002/cb.1453

Huang, C. (2010). Internet use and psychological well-being: A meta-analysis. Cyberpsychology, Behavior, and Social Networking, 13(3), 241–249.

Huang, C. (2017). Time spent on social network sites and psychological well-being: A meta- analysis. Cyberpsychology, Behavior, and Social Networking, 20(6), 346–354.

Johannes, N., Meier, A., Reinecke, L., Ehlert, S., Setiawan, D. N., Walasek, N., Dienlin, T., Buijzen, M., & Veling, H. (2020). The relationship between online vigilance and affective well-being in everyday life: Combining smartphone logging with experience sampling. Media Psychology.

Kim, C., & Shen, C. (2020). Connecting activities on social network sites and life satisfaction: A comparison of older and younger users. Computers in Human Behavior, 105, 106222.

Liu, D., Baumeister, R. F., Yang, C., & Hu, B. (2019). Digital communication media use and psychological well-being: A meta-analysis. Journal of Computer-Mediated Communication, 24(5), 259–273.

Masur, P. K. (2021). Digital Communication Effects on Loneliness and Life Satisfaction. In J. Nussbaum (Ed.), Oxford Research Encyclopedia of Communication. Oxford University Press. https://doi.org/10.1093/acrefore/9780190228613.013.1129

Meier, A., & Reinecke, L. (2020, October 21). Computer-mediated communication, social media, and mental health: A conceptual and empirical meta-review. Communication Research.

Orben, A. (2020b). Teenagers, screens and social media: A narrative review of reviews and key studies. Social Psychiatry and Psychiatric Epidemiology, 55(4), 407–414.

Schemer, C., Masur, P. K., Geiss, S., Müller, P., & Schäfer, S. (2020). The impact of internet and social media use on well-being: A longitudinal analysis of adolescents across nine years. Journal of Computer-Mediated Communication, 26(1), 1–21.

Tromholt, M. (2016). The Facebook experiment: Quitting Facebook leads to higher levels of well-being. Cyberpsychology, Behavior, and Social Networking, 19(11), 661–666. doi:10.1089/cyber.2016.0259

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: PONE-D-21-20386-review.pdf

Click here for additional data file.

14 Mar 2022

We uploaded a document responding point by point to each editor and reviewer comment.

13 Apr 2022

28 Jun 2022

Please see the attached document.

Submitted filename: Response to Reviewers.pdf

Click here for additional data file.

20 Jul 2022

Effects of restricting social media usage on wellbeing and performance: A randomized control trial among students

PONE-D-21-20386R2

Dear Dr. Collis,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Daniel Romer

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

I found two typos and there may be more. i am attaching the pdf with those places highlighted.

Reviewers' comments:

Submitted filename: PONE-D-21-20386_R2[11].pdf

Click here for additional data file.

3 Aug 2022

PONE-D-21-20386R2

Effects of restricting social media usage on wellbeing and performance: A randomized control trial among students

Dear Dr. Collis:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Daniel Romer

Academic Editor

PLOS ONE