Assessing repetitive negative thinking in daily life: Development of an ecological momentary assessment paradigm.

Repetitive negative thinking (RNT) is a transdiagnostic process and a promising target for prevention and treatment of mental disorders. RNT is typically assessed via self-report questionnaires with most studies focusing on one type of RNT (i.e., worry or rumination) and one specific disorder (i.e., anxiety or depression). However, responses to such questionnaires may be biased by memory and metacognitive beliefs. Recently, Ecological Momentary Assessment (EMA) has been employed to minimize these biases. This study aims to develop an EMA paradigm to measure RNT as a transdiagnostic process in natural settings. Based on empirical and theoretical considerations, an item pool was created encompassing RNT content and processes. We then (1) tested model fit of a content-related and a process-related model for assessing RNT as an individual difference variable, (2) investigated the reliability and construct validity of the proposed scale(s), and (3) determined the optimal sampling design. One hundred fifty healthy participants aged 18 to 40 years filled out baseline questionnaires on rumination, worry, RNT, symptoms of depression, anxiety, and stress. Participants received 8 semi-random daily prompts assessing RNT over 14 days. After the EMA phase, participants answered questionnaires on depression, anxiety, and stress again. Multilevel confirmatory factor analysis revealed excellent model fit for the process-related model but unsatisfactory fit for the content-related model. Different hybrid models were additionally explored, yielding one model with satisfactory fit. Both the process-related and the hybrid scale showed good reliability and good convergent validity and were significantly associated with symptoms of depression, anxiety, and stress after the EMA phase when controlling for baseline scores. Further analyses found that a sampling design of 5 daily assessments across 10 days yielded the best tradeoff between participant burden and information retained by EMA. In sum, this paper presents a promising paradigm for assessing RNT in daily life.

Introduction

Repetitive negative thinking (RNT) is a transdiagnostic process that has been shown to play an important role in the development and maintenance of emotional disorders [1-3]. RNT can be defined as a recurrent thinking process that is focused on negative content and perceived as difficult to control. Thus, RNT encompasses different phenomena that have traditionally been studied in isolation, such as worry, depressive rumination, and post-event processing in social anxiety [1]. Recent treatment approaches targeting RNT have yielded promising results, indicating that RNT can be regarded as a proximal risk and maintaining factor that is a suitable target for both prevention and treatment [e.g., 4–6, see also 7]. Due to the relevance of RNT as a transdiagnostic factor, it is of crucial importance that reliable and valid measures of RNT are available for both research and clinical purposes.

So far, RNT is commonly measured as a trait using self-report questionnaires, i.e. asking respondents about the typical content and/or style of their thinking [8-10]. However, the validity of such questionnaires has been called into question. Researchers have argued that self-report questionnaires, especially when referring to a retrospective period (e.g., in the past month) or assessing typical behavior/experience from a global perspective (e.g., general tendencies or traits), reflect a constructed experience that is biased by time, (metacognitive) beliefs, and state factors, rather than reflecting the actual experience or behavior [cf. 11,12].

Discrepancies between retrospective or global reports and real-time assessments of actual experiences have repeatedly been shown by research. For instance, people tend to overestimate internal experiences, such as emotions and physical pain, when asked retrospectively [cf. 11, see 12, for a review]. In the context of RNT, one study found that trait measures of worry only accounted for 24% of the variance of daily-life worry [13]. Furthermore, a recent study showed that momentary rumination predicted higher cortisol levels both in depressed and healthy participants in daily life, whereas trait rumination and retrospectively assessed depressive symptoms failed to predict this effect [14].

Thus, real-time assessments represent a promising approach to capture momentary experiences and contextual information, while minimizing retrospective biases and retaining ecologically valid data [11,15]. Ecological momentary assessment [EMA; 16]–also referred to as the Experience Sampling Method [ESM; 17]–captures individual processes in (near) real-time across multiple time points in a natural environment.

In the recent decade, several studies have employed EMA to assess different forms of momentary RNT [14,18-21]. However, the majority of these studies have predominantly focused on one specific content of RNT only (e.g., either worry, i.e. thinking about potential negative future situations, or rumination, i.e., self-focus on problems and feelings) within the context of specific disorders or symptom dimensions (e.g., either anxiety or depression). To the best of our knowledge, no EMA studies to date have investigated RNT as a transdiagnostic construct by focusing not only on the content but also on the process of RNT across symptom dimensions. Furthermore, even though EMA has received much attention from researchers during the past years thanks to the rapid technological progress, most studies have employed items based on face validity. A systematic investigation into the psychometric properties of the EMA assessment is largely missing [22], which is also the case in the field of RNT. Finally, most EMA studies in this area have focused on assessing state rumination or worry in order to investigate its relationship with, e.g., state changes in mood, cognitions or physiology [e.g., 23,24]. In contrast, the current study aims to develop a brief EMA paradigm to reliably measure the tendency to engage in RNT in daily life as an individual difference variable. To this end, we followed three methodological steps: (1) determining the items of the EMA paradigm, (2) testing the reliability and construct validity of the EMA scale(s), and (3) determining the optimal sampling frequency and duration (i.e., sampling design) of the EMA paradigm. In sum, the present paper presents the development of a reliable and valid EMA paradigm to assess RNT as an individual difference variable in daily life.

Materials and methods

Participants

In total, 150 participants aged 18 to 40 years (M = 22.46, SD = 4.01, 66.8% female) filled out the baseline questionnaires. Participants were included in the study if they had German language skills comparable to a native speaker and were currently not in treatment for mental disorders, as this study aimed for a non-clinical sample. Participants received either course credit or monetary compensation. For monetary compensation, participants received 8 € per hour for appointments and had the chance to win one of four 50 € vouchers depending on their compliance during EMA. For student participants, course credit was also given based on compliance during EMA. Nine participants dropped out due to technical difficulties or personal reasons and one participant was excluded due to being in treatment for a mental disorder. The final sample included 140 individuals for further analysis.

Self-report questionnaires

Rumination, worry, and repetitive negative thinking

Rumination was assessed using the Responses Styles Questionnaire (RSQ-10D) [25,26]. The RSQ-10 is based on the Ruminative Responses Scale (RRS) by Nolen-Hoeksema and Morrow [27], but focuses on items that are not confounded with current symptoms of depression. Items are rated from 1 (“never”) to 4 (“almost always”). A brooding and a reflection subscale can be computed, each consisting of five items. As higher brooding scores have been shown to predict depression, this subscale is assumed to capture dysfunctional rumination [21,26,28]. The RSQ-10 shows high reliability with an internal consistency of Cronbach’s α = .92 for all items, and .75 and .78 for reflection and brooding, respectively [28]. In the current study, internal consistency was α = .69 for the brooding scale.

The Penn State Worry Questionnaire (PSWQ) [29,30] was used to assess trait worry. The PSWQ consists of 16 items measuring pathological worry. Each statement is rated on a 5-point scale (1 = “not at all typical of me”, 5 = “very typical of me”). After recoding five negatively worded items, a total score from 16 to 80 can be achieved with higher scores indicating higher worry levels. Internal consistency ranges from α = .87 to .93 in non-clinical samples and the PSWQ shows good convergent and discriminant validity [31-33]. In the current study, internal consistency was very good with α = .90.

To assess transdiagnostic process characteristics of RNT, the Perseverative Thinking Questionnaire (PTQ) [8] was used. The PTQ comprises three core criteria of RNT, namely repetitiveness, intrusiveness, and difficulty to disengage from negative thoughts. It also measures mental capacity occupied by RNT and subjective unproductiveness of RNT. In total, 15 items are rated from 0 (“never”) to 4 (“almost always”) with a total score ranging from 0 to 60. The PTQ possesses high internal consistency, ranging from α = .92 to .94, and good discriminant and convergent validity [8,34]. In the current study, internal consistency was α = .93.

Stress, anxiety, and depression

Stress was assessed using the Perceived Stress Scale (PSS-10) [35,36]. The scale measures the degree to which participants have perceived their life situations in the past month as stressful, more precisely as uncontrollable, overwhelming and unforeseeable. Higher scores reflect a higher level of perceived stress, as the items are rated from 0 (“never”) to 4 (“very often”), resulting in a maximum score of 40. The 10-item version of the PSS shows a good reliability of α = .78 to.91 [37] and good concurrent validity [38]. In the current study, internal consistency was α = .84.

Generalized anxiety was assessed using the Generalized Anxiety Disorder questionnaire (GAD-7) by Spitzer, Kroenke, Williams, and Löwe [39]. The presence and severity of seven GAD symptoms over the past two weeks are rated on a 4-point scale (0 = “not at all”, 3 = “nearly every day”). This results in a sum score of 21, with a cutoff score for GAD of 10. Internal consistency ranges between α = .82 and .92 in German studies, reflecting high reliability [40,41]. The GAD-7 correlates adequately with depression and self-esteem scales, suggesting construct validity [42]. In the current study, internal consistency was somewhat lower with α = .78 for the GAD-7.

Depression was assessed using the Patient Health Questionnaire–Depression (PHQ-D), [43]. All nine DSM-IV criteria for Major Depressive Disorder are rated as 0 (“not at all”) to 3 (“nearly every day”) regarding their presence in the last two weeks. While the PHQ-D can be used to screen for depression, we used the sum score of the PHQ-D to assess severity of depressive symptoms on a continuum. The PHQ-D has excellent internal reliability, demonstrated by α = .86 to .89 [44]. In the current study, internal consistency was somewhat lower with α = .78 for the PHQ-D.

Additionally, stress, anxiety, and depression was assessed using the Depression-Anxiety-Stress-Scales (DASS), which are suitable for measuring these symptoms in non-clinical populations [45,46]. Forty-two items assess symptoms of depression, anxiety, and stress over the past week on three subscales with a 3-point rating (0 = “did not apply to me at all”, 3 = “applied to me very much, or most of the time”). All scales possess good discriminant and convergent validity and high reliability with α = .91, .78 to .82, and .81 to .89 for depression, anxiety, and stress, respectively [45,47]. In the current study, internal consistency ranged from α = .70 to .79.

Ecological Momentary Assessment (EMA)

Participants received 8 notifications per day, pseudorandomized over a 12-hour time-window starting from 10 AM on weekdays and 12 PM on weekends for 14 consecutive days. After receiving a notification, participants had 15 minutes to start answering 12–14 questions about current mood, emotional events since the last notification, and current content and processes of RNT (see Table 1). Participants were reminded 5 and 10 minutes after the first notification if they had not answered the questions up to this point.

Table 1

EMA paradigm: Items assessing repetitive negative thinking.

RNT Aspect		Item	Scale
Content
1.	Feelings (FEEL)	At the moment I am thinking about my feelings	1: “not at all”, 7: “very much”
2.	Problems (PROB)	At the moment I am thinking about my problems	1: “not at all”, 7: “very much”
3.	Past(PAST)	At the moment I am thinking about upsetting memories	1: “not at all”, 7: “very much”
4.	Future (FUT)	At the moment I am thinking about negative future situations	1: “not at all”, 7: “very much”
Process
5.	Duration (DUR)	How long have you been thinking about these topics up to this moment?	1: “not at all”, 7: “more than 120 min.”
6.	Distress (BUR)	How much do you feel weighed down by these thoughts at this moment?	1: “not at all”, 7: “very much”
7.	Repetitiveness (RPT)	The same thoughts keep going through my mind again and again.	1: “not at all”, 7: “very much”
8.	Intrusiveness (INTR)	Thoughts come to my mind without me wanting them to.	1: “not at all”, 7: “very much”
9.	Uncontrollability (CTRL)	I get stuck on certain issues and can't move on.	1: “not at all”, 7: “very much”

Momentary repetitive negative thinking

In order to determine the items for the EMA paradigm, a literature research was conducted to identify concepts and items used in earlier studies to assess RNT.

Content. Four items were used to assess the content of RNT. The first two items asked respondents to indicate how much they are currently thinking about their feelings and problems, respectively. The items were modeled after similar items used in earlier studies [21], but phrased as whole sentences to fit with the format of the EMA assessment in this study. A composite score of both items has been used in various studies, showing good reliability and validity [14,21,23,48,49]. Worry and rumination are the two most frequently studied forms of RNT, and have been shown to share the same processes while differing in their temporal orientation [2,50]. In order to represent both forms of RNT in the EMA scale, we further included two items (thinking about upsetting events/ negative future situations) modeled after similar items originally introduced by Selby and colleagues [20]. An average score of all four items was significantly associated with trait rumination. Within-person reliability for these items ranged from R = .89 to .95 [20]. These four items made up the content-related model.

Process. Whereas RNT has traditionally been measured by asking participants about the content of recurrent thinking (as also reflected in the EMA items described above), recent transdiagnostic concepts of RNT put a stronger emphasis on characteristics of thinking process during RNT that is suggested to be transdiagnostic, i.e., similar across diagnostic categories [1,51]. An alternative way to measure RNT, therefore, focuses on assessing these process characteristics. For example, the Perseverative Thinking Questionnaire [8] was developed as a trait questionnaire for this purpose. In this study, three items were adapted from the PTQ to assess repetitiveness, intrusiveness, and difficulty to disengage from negative thoughts as the core characteristics of RNT. The latter, also labeled as uncontrollability, has been used in other EMA studies showing a significant association with trait rumination and prolonged sleep onset latency [52,53]. All three items showed significant factor loadings between .81 and .85 onto one factor in the validation study. Amongst others, this factor was significantly related to the brooding subscale of the RSQ, pathological worry, as well as symptoms of depression and anxiety [8,54,55].

Furthermore, it is assumed that both subjective burden and duration of RNT capture maladaptive processes as they indicate distress and failure to stop negative thoughts [13,18,56,57]. Therefore, one item assessing distress (subjective burden) caused by RNT and one item assessing duration of RNT were adapted from the work of Thielsch and colleagues [18]. The latter item was designed to assess duration of RNT between the previous and current notification, as participants received a notification about every two hours.

In addition to RNT items, participants also answered EMA questions regarding current mood and emotional events experienced, which are not relevant for the analyses of this paper.

Procedure

Participants were recruited via posters, online announcements in social media and mailing lists. At the first appointment, the investigator explained the purpose and procedure of the study. After providing informed consent, participants were introduced to the EMA app (“Tellmi”, developed at LMU Munich for research use), which they had installed on their smartphone (iOS or Android) prior to the appointment. If they did not own a smartphone, or had technical difficulties with the app on their smartphone, participants received a smartphone for the duration of the study. Participants were given the opportunity to look at all EMA questions and ask questions regarding app use and procedure. Finally, participants filled out demographic information and the PHQ-D, GAD-7, DASS, PSS, PSWQ, RSQ-10, and PTQ. The EMA period started on the day after the first appointment and lasted 14 days. After this period, participants had a second appointment to again fill out the PHQ-D, GAD-7, DASS and PSS regarding the past weeks. Additionally, a semi-structured interview was conducted to assess acceptability and feasibility of the EMA paradigm regarding duration, frequency and items.

The study was approved by the local ethics committee and is in accordance with the Declaration of Helsinki.

Statistical analysis

Statistical analyses were conducted in R, version 3.4.3 [58]. To test if the content-related and process-related model fit the observed data, we conducted multilevel confirmatory factor analyses (MCFA) starting with the pre-determined sets of items. Model fit was estimated using the R-package lavaan [59], following the procedure described in Huang [60]. As our data had a two-level nested structure (i.e., the occasion level nested into the person level), we assumed the same unifactor structure at each of the levels. A MCFA model is typically specified on decomposed within- and between-person covariance matrices, which allows us to consider both levels simultaneously in a single model.

Two approaches were investigated, one using all four content-related items (i.e., thinking about feelings, problems, past-oriented rumination, future-oriented worry). The second approach comprised the process-related items assessing repetitiveness, intrusiveness, uncontrollability, and burden of RNT. One item assessing duration of RNT was excluded from MCFA analyses based on participant feedback, which indicated that participants had difficulties with answering this item. For each model, goodness-of-fit indices were computed to test whether the model (representing a latent concept of RNT each at the between- and within-person level) fits the data. Goodness-of-fit was evaluated based on the following cutoff criteria: Comparative-Fit Index (CFI) ≥ .95 Root Mean Square Error of Approximation (RMSEA) ≤ .06, Standardized Mean Square Error of Approximation (SRMR) ≤ .08 [61].

Reliability coefficients for multilevel data were calculated for the yielded EMA scale(s) following the generalizability theory (GT) approach [62,63]. Whereas classical test theory decomposes the variance of a given observed score into a true variance and an error variance, GT takes into account several different sources of variance, allowing for the estimation of the reliability of within-person changes (called R). Furthermore, the GT framework can be used to estimate the reliability of a scale on a randomly selected day (R) and the reliability of a scale across all days (R), i.e., the between-person reliability.

To optimize the EMA paradigm in terms of the sampling design, different subsets of data were created for different frequencies (number of notifications per day) and durations (number of days), starting with the lowest acceptable frequency (3 notifications per day) and the lowest acceptable duration (3 days) and ending with the highest possible frequency and duration (i.e., 8 notifications per day for 14 days), which constituted the total set of observations. For frequencies lower than 8, we tried to choose notifications that were equally distributed throughout the day, i.e., for a frequency of 3 notifications per day, we chose to include the 1st, 4th, and 7th notification of the day into the subset. For durations lower than 14 days, we chose a range of subsequent days, always starting from day 1.

Three times per day was determined as the lowest acceptable frequency to account for variations in RNT within the day. Also, a duration of three days was determined as the minimum number to account for day-to-day variations and to increase reliability of daily behavior.

Correlations were calculated between each subset of observations and the total set of observations. Correlations were calculated for different person-level parameters, such as mean, variability, and instability (root mean square successive difference; RMSSD). This resulted in a 12 (number of days: 3–14) x 6 (frequency per day: 3–8) matrix for each parameter. Data were visually inspected to find a reasonable balance between information gain and participant burden, while also accounting for participant feedback from a semi-structured interview at the end of the study to consider acceptability.

Results

Data cleaning and compliance

Out of 15,680 possible observations (140 persons x 14 days x 8 assessments per day), 11,673 (76.6%) were completed by participants. Ninety-seven observations (i.e., < 0.01% of total) were deleted due to technical errors. All participants with a response rate of less than 40% were excluded (n = 9). For plausibility, person-level standard deviations were checked for each item across the EMA period. Participants with a standard deviation of 0 in at least one RNT item were excluded (n = 11). Therefore, 120 participants aged 18 to 40 years (M = 22.25, SD = 3.89, 71% female, n = 118) with 10,498 observations remained for data analysis.

Descriptive data of EMA items

EMA scores were first aggregated at the person level, which were further used to calculate descriptive statistics (see Table 2). Grand means for RNT items ranged from 1.81 for thinking about upsetting memories (PAST), to 2.25 for thinking about negative future situations (PROB). This is comparable to other non-clinical samples [14].

Table 2

Grand means of EMA items.

Name	M	SD	Mdn	Min	Max	Range
1. FEEL	2.15	0.75	2.14	1.03	4.15	3.12
2. PROB	2.23	0.80	2.09	1.03	4.04	3.01
3. PAST	1.81	0.68	1.63	1.01	3.72	2.71
4. FUTR	2.25	0.89	2.08	1.03	4.73	3.70
5. DUR	2.00	0.73	1.84	1.01	4.33	3.32
6. BUR	1.96	0.75	1.78	1.01	4.08	3.07
7. RPT	2.16	0.95	1.97	1.01	5.02	4.01
8. INTR	2.06	0.98	1.72	1.01	4.94	3.93
9. CTRL	2.20	1.06	1.93	1.01	5.83	4.82

N = 120; FEEL: feelings; PROB: problems; PAST: upsetting memories; FUTR: negative future situations; DUR: duration; BUR: subjective burden; RPT: repetitiveness; INTR: intrusiveness; CTRL: uncontrollability; EMA: Ecological Momentary Assessment.

Inspection of the correlation between different RNT items showed that there were high inter-correlations between the content-related items (FEEL, PROB, PAST, FUTR), with the highest correlation between thinking about problems (PROB) and thinking about negative future situations (FUTR; r = .92) and the lowest correlation between thinking about negative future situations (FUTR) and thinking about upsetting memories (PAST; r = .71) on a between-person level (see Table 3). Very high correlations on a between-person level were found between the three PTQ items that capture RNT processes (RPT, INTR, CTRL), ranging from r = .93 to .95. These items also correlated highly on a within-person level, ranging from r = .69 to .70.

Table 3

Between- and within-person correlations of RNT items.

	1. FEEL	2. PROB	3. PAST	4. FUTR	5. DUR	6. BUR	7. RPT	8. INTR	9. CTRL
1. FEEL	1.00	0.88	0.80	0.78	0.75	0.89	0.70	0.67	0.63
2 PROB	0.56	1.00	0.76	0.92	0.77	0.90	0.74	0.68	0.68
3. PAST	0.54	0.50	1.00	0.71	0.65	0.81	0.63	0.64	0.57
4. FUTR	0.49	0.68	0.47	1.00	0.72	0.86	0.74	0.67	0.68
5. DUR	0.46	0.52	0.44	0.51	1.00	0.75	0.58	0.51	0.50
6. BUR	0.57	0.65	0.51	0.62	0.53	1.00	0.78	0.73	0.74
7. RPT	0.53	0.55	0.47	0.54	0.50	0.62	1.00	0.95	0.95
8. INTR	0.49	0.52	0.46	0.52	0.46	0.62	0.69	1.00	0.93
9. CTRL	0.50	0.57	0.47	0.55	0.48	0.62	0.71	0.70	1.00

Above the diagonal = between-person correlations; below the diagonal = within-person correlations; n(within) = 10355, n(between) = 120; FEEL: feelings; PROB: problems; PAST: upsetting memories; FUTR: negative future situations; DUR: duration; BUR: subjective burden; RPT: repetitiveness; INTR: intrusiveness; CTRL: uncontrollability; RNT: repetitive negative thinking.

Testing different RNT models

Based on the two approaches of selecting the items for our scale, two different models were tested comprising four items each: (a) a content-related approach, including the commonly used items of ruminative self-focus (FEEL, PROB), extended by two items that reflect the temporal dimension (PAST, FUTR) and (b) a process-related approach including the three process variables adapted from the PTQ (RPT, INTR, CTRL), extended by subjective burden/distress regarding RNT (BUR).

a) Content-related model and hybrid model

The model including all four content-related items yielded an unacceptable model fit (RMSEA = .174, see Table 4). To explore whether a satisfactory model could be found that retained the two original content items (FEEL, PROB), six different 4-item hybrid models were tested retaining FEEL and PROB in each of the models and testing all combinations with the remaining four process-related items. Two models were found to exhibit acceptable model fit: one model encompassing subjective burden (BUR) and uncontrollability of RNT (CTRL) (RMSEA = .047), another model encompassing subjective burden (BUR) and repetitiveness of RNT (RPT) (RMSEA = .059) in addition to both content items. We chose the former model including CTRL for further analyses, as it yielded the best model fit. MCFA results for all tested hybrid models are shown in S1 Table.

Table 4

Multilevel confirmatory factor analysis results for a content-related, a process-related, and an exploratory hybrid model.

	χ2	df	CFI	SRMR	RMSEA	90% CIL1	90% CIU1
a) Content-related model
Original model
FEEL, PROB, PAST, FUTR	636.01***	4	.960	.040	.174	.162	.185
	W: 610.30
	B: 25.71
Exploratory hybrid model
FEEL, PROB, BUR, CTRL	50.10***	4	.997	.009	.047	.036	.059
	W: 40.57
	B: 9.53
b) Process-related model
Original model
RPT, INTR, CTRL, BUR	9.82*	4	1	.003	.017	.003	.030
	W: 7.41
	B: 2.40

1Upper/lower confidence interval for RMSEA;

*** p < .001;

* p < .05;

n(within) = 10355, n(between) = 120. FEEL: feelings; PROB: problems; PAST: upsetting memories; FUTR: negative future situations; BUR: subjective burden; RPT: repetitiveness; INTR: intrusiveness; CTRL: uncontrollability; CFI: Comparative-Fit Index; SRMR: Standardized Mean Square Error of Approximation; RMSEA: Root Mean Square Error of Approximation; W: within-person level; B: between-person level. Final models are highlighted in bold.

b) Process-related model

The second approach included all three adapted PTQ items which assessed core processes of RNT plus a single item to assess subjective burden of RNT. The model showed excellent model fit (RMSEA = .017).

Further analyses were conducted regarding the two scales with good model fit, i.e. the process-related and the exploratory hybrid model.

Psychometric information

Reliability

Between-person reliability for both the process-related and hybrid model was excellent with R > .99. When looking at within-person reliability, the hybrid model also showed a good reliability of R = .84; which was lower than the reliability of the process model with R = .88. When calculating reliability for a random day, reliability was R = .44 for the hybrid model and R = .50 for the process-related model.

Convergent validity

Small to moderate correlations were found between RNT assessed via EMA and trait questionnaires assessing rumination, worry, and RNT (see Table 5). Similarly, small to moderate correlations were found between the two EMA scales and symptom scores at baseline.

Table 5

Correlations between EMA scales, trait measures, and symptom measures.

	Trait measures			Psychopathology
	PSWQ	RSQ-b	PTQ	PHQ-D	GAD-7	PSS	DASS-D	DASS-A	DASS-S
EMA-RNT (hybrid)	.32***	.20*	.36***	.31***	.38***	.36***	.38***	.34***	.24*
EMA-RNT (process)	.30***	.16	.37***	.30***	.33***	.29***	.32***	.37***	.24*

*** p < .001;

* p < .05;

N = 118; EMA; ecological momentary assessment; RNT: repetitive negative thinking; PSWQ: Penn-State Worry Questionnaire; RSQ-b: Response Styles Questionnaire—brooding; PTQ: Perseverative Thinking Questionnaire; PHQ-D: Patient Health Questionnaire—Depression; GAD-7: Generalized Anxiety Disorder Questionnaire; PSS: Perceived Stress Scale; DASS: Depression-Anxiety-Stress-Scales.

Furthermore, regression analyses were conducted to inspect associations between EMA-assessed RNT and symptom scores at follow-up. Symptoms of depression, anxiety, and stress after two weeks were significantly predicted by both EMA scales, after controlling for baseline scores for depression, anxiety, and stress, respectively (see Table 6). This finding indicates that both scales have high concurrent validity.

Table 6

Prediction of depression, anxiety, and stress symptoms by the two EMA scales.

	Depression Post (DASS)				Anxiety Post (DASS)				Stress Post (DASS)
	B	SE	β	95% CI	B	SE	β	95% CI	B	SE	β	95% CI
a) Hybrid model
Intercept	-0.44	.64	0.00	-1.71–0.83	-0.89	.43	0.00*	-1.74 –-0.05	-0.57	.79	0.00	-2.13–0.99
Baseline symptoms	0.48	.08	0.49***	0.32–0.64	0.57	.06	0.62***	0.45–0.69	0.53	.08	0.52***	0.38–0.68
EMA-RNT (hybrid)	0.22	.08	0.23**	0.06–0.37	0.20	.05	0.27***	0.10–0.31	0.29	.09	0.26**	0.12–0.47
	Adjusted R2 = .37, F(2,109) = 33.65***				Adjusted R2 = .57, F(2,109) = 73.48***				Adjusted R2 = .40, F(2,109) = 37.69***
b) Process-related model
Intercept	0.08	.57	0.00	-1.04–1.20	-0.56	.37	0.00	-1.30–0.18	0.32	.72	0.00	-1.10–1.75
Baseline symptoms	0.51	.08	0.52***	0.36–0.67	0.57	.06	0.62***	0.44–0.69	0.55	.08	0.54***	0.39–0.71
EMA-RNT (process)	0.15	.07	0.19*	0.01–0.28	0.17	.05	0.25***	0.08–0.26	0.18	.08	0.18*	0.02–0.33
	Adjusted R2 = .35, F(2,109) = 31.52***				Adjusted R2 = .56, F(2,109) = 71.50***				Adjusted R2 = .37, F(2,109) = 33.11***

*** p < .001;

** p < .01;

* p < .05;

DASS: Depression-Anxiety-Stress-Scales; EMA: ecological momentary assessment; RNT: repetitive negative thinking.

Sampling design of the EMA paradigm

Correlations of person means between subsets of observations varying in sampling design (frequency of daily assessments, duration of EMA phase) and all observations were calculated for both EMA scales using different statistical values (i.e., mean, standard deviation, and RMSSD). Correlations of person means are shown in Fig 1 for the hybrid EMA scale and in Fig 2 for the process-related EMA scale. Correlations of all person-level parameters are reported in S1 File.

Fig 1

Hybrid EMA scale.

Correlations of person means between subsets of observations and all observations, varying in frequency between 3 and 8 times per day and duration between 3 and 14 days; EMA: ecological momentary assessment.

Fig 2

Process-related EMA scale.

Correlations of person means between subsets of observations and all observations, varying in frequency between 3 and 8 times per day and duration between 3 and 14 days; EMA: ecological momentary assessment.

At the maximum frequency of 8 daily assessments, the correlation between person means estimated on the subset and total set of observations exceeded .95 after 8 days. This extent of correlation between person means was reached for frequencies higher than 3 times per day at an EMA duration of at least 10 days across both scales. That is, a subset of observations with a sampling design of 4 notifications per day and a duration of 10 days explains sufficient variance of person means based on all observations.

Considering that about 20% of notifications are missed by participants, a duration of 10 days with a frequency of 5 times per day was chosen as an optimal tradeoff between participant burden and information gain. Correlations between subset and total set of observations ranged from r = .82 to .90 for person-level variability and instability across both scales when sampling frequency was fixed at 4 times per day for 10 days (i.e. a sampling design of 5 times per day for 10 days assuming 20% of missing information).

This result was further endorsed by participant feedback, indicating that the frequency during the day appears to exert higher burden than the total duration of days: while 53.5% stated that the frequency of assessments was too high, only 13.2% indicated that the assessment duration of 14 days was too long. According to participant feedback, a frequency of 4 to 5 times per day was perceived as acceptable: 66.6% of participants who answered this question preferred a frequency of 5 or 6 times per day, while another 23.3% of participants stated that 4 times per day was their favored frequency.

Discussion

The aim of this paper was to develop a reliable EMA paradigm to assess RNT in daily life. To achieve this, we followed three steps: (1) determining the items to be included in the EMA paradigm, (2) examining reliability and construct validity of the EMA scale(s), and (3) optimizing the sampling design while taking into account participant burden.

To determine the optimal combination of items included in the EMA paradigm, we tested a content-related and a process-related scale. Whereas the process-related model showed excellent model fit according to MCFA, the content-related model yielded unacceptable model fit. To test, whether an acceptable model could be found retaining the two original content items of momentary thinking about problems and feelings, we explored different 4-item hybrid models, each including both original content items and two process-related items. An exploratory hybrid model including difficulty to disengage from thoughts (uncontrollability) and subjective burden (distress) of RNT was found to yield the best model fit.

Next, we investigated reliability and convergent validity of both scales. Both, the process-related and hybrid scale showed good reliability on a between-person level as well as a within-person level, with the process-related scale exhibiting slightly higher reliability. Both scales showed low to moderate associations with standard trait questionnaires on rumination, worry, RNT, and symptom levels of depression, anxiety, and stress. Low to moderate correlations between EMA-assessed rumination or worry and trait measures have also been reported by other studies [14,20,64]. A reason for low correlations between trait measures and daily life RNT might be the different methods in assessing RNT. Whereas EMA aims to reduce retrospective biases and increase ecological validity by assessing RNT in the natural setting in near real-time, trait measures might capture more generalized (metacognitive) beliefs about RNT, rather than the behavior itself.

Additionally, we found that both scales were able to significantly predict symptoms of depression, anxiety, and stress at follow-up over and above baseline symptoms. Since the DASS refers to symptoms within the past week and the follow-up assessment took place shortly after the two week EMA phase, there is an overlap in the time window measured by EMA and by the retrospective DASS. Future studies are needed to determine whether measuring RNT via EMA can significantly predict these symptoms after a longer time interval and whether it outperforms traditional trait questionnaires measuring RNT.

To determine the optimal number of assessment days and frequency of assessments per day, we both analyzed participant feedback based on a semi-structured interview at the end of the EMA phase and computed correlations between subsets of observations and all observations for different sampling designs. A good tradeoff between participant burden and information gain was found at a frequency of five daily assessments for ten consecutive days, allowing for a rate of 20% missing data. Generally, we found that about 40 to 50 observations per participant seemed sufficient to capture the variability and average score of RNT. That is, this number of observations appeared to adequately approximate the person means estimated from all observations.

In sum, the findings of the current paper suggest two possible 4-item EMA scales to assess RNT in daily life, which showed excellent model fit, high reliability within and across persons, as well as significant associations with symptom levels of depression, anxiety, and stress. Both scales share two process-related items, namely, distress (subjective burden of RNT) and uncontrollability (difficulty to disengage from thoughts). It may be that distress/burden and uncontrollability are associated with RNT being experienced negatively. For instance, another study showed that unpleasantness and uncontrollability of RNT were particularly strongly associated with negative affect [52]. Uncontrollability was further associated with longer latencies to fall asleep [53]. Although subjective burden of RNT has not received much interest in previous studies, it seems to be of high importance, as it reflects psychological strain and distress, which is a clinically relevant aspect across mental disorders. The only difference between both scales was a purely process-related focus versus an integrated approach using both process- and content-related items. Future studies need to cross-validate findings to investigate whether both scales remain to show high reliability and good model fit in other samples or whether one scale exhibits superiority over the other.

To the best of our knowledge, this is the first study to establish an EMA paradigm measuring RNT from a transdiagnostic perspective. Previous studies have either focused on rumination in the context of depression or on worry in the context of anxiety disorders. However, recent research has shown that both rumination and worry share the same underlying process [51]. In order to assess RNT transdiagnostically, it therefore appears important to include items assessing the process of RNT. Of note, studies examining the differences between content-related versus process-related trait questionnaires regarding their predictive utility for depression and anxiety showed that an underlying RNT factor explained more variance in predicting anxiety and depression than disorder-specific cognitive content (i.e., rumination or worry) [65]. It remains to be tested whether the same applies to RNT assessed via EMA. Future studies should compare the two scales identified in the current study regarding the power to predict future symptom levels. As the two scales differ regarding their inclusion of content-related items, this will not only be informative from a purely psychometric but also from a theoretical point of view.

Furthermore, we believe that this is the first study closely investigating the psychometric properties of EMA items to assess RNT. While some previous studies have in part reported reliabilities or concurrent validity of their used scale, unfortunately, this seems to be rather the exception than the rule. Thus, leading researchers in the EMA field are now calling for common standards in study design and selection of items to improve quality of EMA studies and facilitate replication [22].

On the other hand, some limitations are noteworthy. First, this study was not able to determine the predictive validity of the developed EMA paradigm. Consistent with findings of studies assessing RNT via trait questionnaires, a next step in validating the present EMA paradigm is to investigate whether EMA-assessed RNT significantly predicts symptoms of anxiety and depression after a longer time interval. In addition to examining whether an aggregated score of RNT across the EMA phase significantly predicts psychopathology, other dynamic measures can be taken into account. For instance, one study has shown that EMA-assessed instability of rumination predicts depressive symptoms after six and 36 months over and above average levels of daily-life rumination [66]. Whereas the underlying causes for this finding remain to be investigated, the authors speculated that daily life stressors may be related to higher fluctuations in rumination, resulting in higher depression levels. Therefore, not only instability of RNT, but also stressors and the interaction of RNT with stress in daily life might be valuable targets for further investigation [67].

Furthermore, our proposed frequency and duration of the EMA paradigm has to be cross-validated in a future study, since our findings are based on post-hoc analyses without actually comparing different frequencies and durations in vivo. Moreover, the sampling design of EMA studies is always dependent on the base rate or temporal variability of the measured construct. Therefore, our finding cannot be generalized to EMA studies overall, but is confined to the measurement of RNT.

Last, we assessed RNT in a non-clinical sample. Therefore, we excluded participants which indicated to be suffering from a mental disorder. Inspection of the GAD and depression scores showed that 15 participants scored in the range for clinically significant levels of generalized anxiety (≥ 10), while 13 participants exceeded the cutoff score (≥ 10) for moderate levels of depressive symptoms. In general, the majority of the sample can be labeled as healthy participants, which is also reflected in the low mean scores of RNT. The results of this paper are, therefore, confined to non-clinical populations and further research is necessary to investigate whether the same paradigm can be replicated in clinical populations.

Nevertheless, this study proposes two promising EMA scales for assessing RNT in daily life, which are subject to further investigations. With this study, we hope to encourage other researchers to also consider a process-oriented, transdiagnostic perspective on rumination and worry.

Multilevel confirmatory factor analysis results for different exploratory hybrid models.

(DOCX)

Click here for additional data file.

Correlations between subsets of observations and all observations for varying sampling designs regarding the hybrid EMA scale and the process-related EMA scale.

(XLSX)

Click here for additional data file.

22 Jan 2020

PONE-D-19-32809

Assessing repetitive negative thinking in daily life: Development of an ecological momentary assessment paradigm

PLOS ONE

Dear Dr Rosenkranz,

Thank you for submitting your manuscript to PLOS ONE. I was fortunate to have an expert in the field review this manuscript. I have had some difficulty finding a second reviewer, however the first reviewer's review was thorough enough that I am able to make a decision based on that. As you will see below, the reviewer found your paper to have much merit, especially in the novel EMA design. Despite this, the reviewer also found some issues with the paper that are addressable in a revision. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The full reviews are pasted below my signature and as you will see, many of the comments are in regard to the need to provide more clarity around your participant selection process and reduce some redundancy in the introduction.

We would appreciate receiving your revised manuscript by Mar 07 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Evan M Kleiman

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that Table 2 and File S1 in the Supporting Information may include questionnaire items that may have been previously published. The reproduction of previously published work has implications for the copyright that may apply to these publications. We would be grateful if you could clarify whether you have obtained permission from the original copyright holder to republish these items under a CC BY license. If you have not obtained permission to publish these items please remove them from your manuscript. You may wish to replace the text you have removed with relevant question numbers/ brief descriptions of each item; please be sure to include any relevant references and in-text citations.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a well-written and thorough paper attempting to establish a psychometrically-sound, transdiagnostic EMA measure of repetitive negative thinking. It is an important addition to the literature, as the authors rightfully point out that most EMA studies are using untested questionnaires- therefore, moving forward, researchers could employ the items described by the authors and be able to cite the work they have done here to establish reliability/validity/model fit.

My concerns with the manuscript in its current form are as follows:

1. Introduction: Much of the info at the end of the introduction gets repeated in the methods section. It seems to belong fully in the methods and can be cut from the intro. The introduction should frame the question at hand, which it does very nicely up to line 90 on p. 4. From line 91 on, you go into a fairly detailed account of your methods. I would work on cutting a lot of this out/moving info to the methods section if it is not already described there. Retaining big picture concepts (e.g. your discussion of content vs process) is important, but description of item-level info can be saved for the methods.

2. Participants: participants were excluded if they were in treatment for a mental disorder. Please note why this was an exclusionary criterion. Were you attempting to have a non-clinical sample? As we know rates of mental health treatment are low, it is possible that some in your sample still will have scored within clinical ranges on measures of anxiety/depression (eg GAD 7, PHQ D)- please report this information. The attempt to exclude more symptomatic participants in the current study should also be noted in the limitations/future directions section of your discussion- the means on your EMA items seem fairly low across the board, so it is worth noting whether your findings may have been different, including model fit, in a more clinical sample.

3. “Ecological Momentary Assessment”, p. 9 says the 12 hour window on weekends started at 12AM- should this read PM?

4. Statistical analysis: I have not used MCFA so cannot comment on the appropriateness/conduct of these analyses, but feel that this section is well-written

5. P.13. I’m still a little fuzzy on how you calculated subsets. Did you randomly select a given # of responses per day for a participant? And was it only for some days out of the full 14 day period? I would suggest rewording and maybe asking someone unfamiliar with the study to read this section and see if they understand the procedure.

6. Tables 3 and 4- nb and nw phrasing may be confusing to the reader, I suggest spelling out nbetween, nwithin

7. Table 6- looks like the DASS section doesn’t have the beta symbol and instead has a B

8. Discussion: “low to moderate correlations between EMA-assessed measures & trait measures have been reported by other studies.” Expand a bit here on why this might be the case- do you think state measures are more valid? You bring this up in the intro, but don’t bring it back into the discussion.

9. This study finds that the process items tended to fare better than the content items. What might this mean? For instance, you write “Therefore, adding process-related variables to ensure a content independent measurement of RNT in daily life is crucial to investigate RNT as a transdiagnostic process across mental disorders.” Does this mean that ideally content-related items shouldn’t be included at all? More discussion of the implications of your findings would be helpful in the discussion.

10. “In addition to examining whether an aggregated score of RNT across the EMA phase significantly predicts psychopathology, other dynamic measures can be taken into account, such as instability or stress-reactive RNT.” Flesh this idea out a bit more- I didn’t understand how the second half of the sentence related to the first.

11. Supporting table 2 shows that FEEL, PROB, RPT, BUR has an RMSEA of .059. I am not expert in RMSEA but read that values below .06 indicated good fit. In which case, should this be noted in the body of your paper?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

7 Mar 2020

We would like to thank the editor and the reviewer for their thoughtful comments, which have helped us improve the manuscript. We have carefully checked every comment and hope to have answered your comments in a satisfactory manner. Please find our rebuttal below.

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

RESPONSE: We have carefully reviewed the style requirements and have adapted our style where needed.

2. We note that Table 2 and File S1 in the Supporting Information may include questionnaire items that may have been previously published. The reproduction of previously published work has implications for the copyright that may apply to these publications. We would be grateful if you could clarify whether you have obtained permission from the original copyright holder to republish these items under a CC BY license. If you have not obtained permission to publish these items please remove them from your manuscript. You may wish to replace the text you have removed with relevant question numbers/ brief descriptions of each item; please be sure to include any relevant references and in-text citations.

RESPONSE: S1 has now been removed. The RNT items are provided in Table 1 within the manuscript. All other items used during the EMA assessment are briefly described within the text.

As to the RNT items, we have obtained permission to republish the items for Duration, Burden/Distress, Repetitiveness, Intrusiveness, and Uncontrollability. The items for RNT focused on feelings, problems, past, and future were modelled after items used in earlier studies (as described in the manuscript), but did not use the exact phrasing used in these earlier studies. This means that we are not republishing content from earlier manuscripts in these cases. We have now made this aspect clearer in the manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

________________________________________

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

________________________________________

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

________________________________________

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

________________________________________

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a well-written and thorough paper attempting to establish a psychometrically-sound, transdiagnostic EMA measure of repetitive negative thinking. It is an important addition to the literature, as the authors rightfully point out that most EMA studies are using untested questionnaires- therefore, moving forward, researchers could employ the items described by the authors and be able to cite the work they have done here to establish reliability/validity/model fit.

My concerns with the manuscript in its current form are as follows:

1. Introduction: Much of the info at the end of the introduction gets repeated in the methods section. It seems to belong fully in the methods and can be cut from the intro. The introduction should frame the question at hand, which it does very nicely up to line 90 on p. 4. From line 91 on, you go into a fairly detailed account of your methods. I would work on cutting a lot of this out/moving info to the methods section if it is not already described there. Retaining big picture concepts (e.g. your discussion of content vs process) is important, but description of item-level info can be saved for the methods.

RESPONSE: Thank you for this comment. We have now merged parts of the detailed introduction after line 91 with the EMA item description in methods. As a result, we believe that the introduction is now more straightforward and focuses on presenting the bigger picture.

2. Participants: participants were excluded if they were in treatment for a mental disorder. Please note why this was an exclusionary criterion. Were you attempting to have a non-clinical sample? As we know rates of mental health treatment are low, it is possible that some in your sample still will have scored within clinical ranges on measures of anxiety/depression (eg GAD 7, PHQ D)- please report this information. The attempt to exclude more symptomatic participants in the current study should also be noted in the limitations/future directions section of your discussion- the means on your EMA items seem fairly low across the board, so it is worth noting whether your findings may have been different, including model fit, in a more clinical sample.

RESPONSE: We aimed for a non-clinical sample because the development of the EMA paradigm was part of a grant-funded project focusing on prevention. This means that within this project our EMA paradigm will be applied to a non-clinical sample. We have added this part in the paper (p. 5):

Participants were included in the study if they had German language skills comparable to a native speaker and were currently not in treatment for mental disorders, as this study aimed for a non-clinical sample

We have included this issue as a limitation in the discussion part (p. 22):

Last, we assessed RNT in a non-clinical sample. Therefore, we excluded participants who indicated to be suffering from a mental disorder. Inspection of the GAD and depression scores showed that 15 participants scored in the range for clinically significant levels of generalized anxiety (≥ 10), while 13 participants exceeded the cutoff score (≥ 10) for moderate levels of depressive symptoms. In general, the majority of the sample can be labeled as healthy participants, which is also reflected in the low mean scores of RNT. The results of this paper are, therefore, confined to non-clinical populations and further research is necessary to investigate whether the same paradigm can be replicated in clinical populations.

3. “Ecological Momentary Assessment”, p. 9 says the 12 hour window on weekends started at 12AM- should this read PM?

RESPONSE: Yes, we have corrected this error.

4. Statistical analysis: I have not used MCFA so cannot comment on the appropriateness/conduct of these analyses, but feel that this section is well-written

RESPONSE: Thank you. As described, the fine details of MCFA can be consulted in the paper by Huang, 2017, which we have cited.

5. P.13. I’m still a little fuzzy on how you calculated subsets. Did you randomly select a given # of responses per day for a participant? And was it only for some days out of the full 14 day period? I would suggest rewording and maybe asking someone unfamiliar with the study to read this section and see if they understand the procedure.

RESPONSE: We have rephrased this paragraph so that the procedure will become clearer for the reader. As described, the number of notifications per day were “equally spaced throughout the day”, i.e., for a frequency of 3 notifications, we would have used e.g. the 1st, 4th, and 7th notification. We have also included this clarification in the manuscript (p. 14).

6. Tables 3 and 4- nb and nw phrasing may be confusing to the reader, I suggest spelling out nbetween, nwithin

RESPONSE: We have changed this in the tables.

7. Table 6- looks like the DASS section doesn’t have the beta symbol and instead has a B

RESPONSE: Thank you for noticing this error. We have corrected it.

8. Discussion: “low to moderate correlations between EMA-assessed measures & trait measures have been reported by other studies.” Expand a bit here on why this might be the case- do you think state measures are more valid? You bring this up in the intro, but don’t bring it back into the discussion.

RESPONSE: We now bring this issue back in the discussion (p. 20):

A reason for low correlations between trait measures and daily life RNT might be the different methods in assessing RNT. Whereas EMA aims to reduce retrospective biases and increase ecological validity by assessing RNT in the natural setting in near real-time, trait measures might capture more generalized (metacognitive) beliefs about RNT, rather than the behavior itself.

9. This study finds that the process items tended to fare better than the content items. What might this mean? For instance, you write “Therefore, adding process-related variables to ensure a content independent measurement of RNT in daily life is crucial to investigate RNT as a transdiagnostic process across mental disorders.” Does this mean that ideally content-related items shouldn’t be included at all? More discussion of the implications of your findings would be helpful in the discussion.

RESPONSE: In the revised manuscript, we now elaborate on this issue. Specifically, we (1) now argue that process-related items need to be included to assess RNT transdiagnostically, (2) refer to earlier research using trait questionnaires showing that process-related measures are better at predicting psychopathology than content-related ones, but (3) acknowledge that it is ultimately an empirical question whether the same applies to assessing RNT using EMA. The paragraph on p. 21 now reads:

To the best of our knowledge, this is the first study to establish an EMA paradigm measuring RNT from a transdiagnostic perspective. Previous studies have either focused on rumination in the context of depression or on worry in the context of anxiety disorders. However, recent research has shown that both rumination and worry share the same underlying process [50]. In order to assess RNT transdiagnostically, it therefore appears important to include items assessing the process of RNT. Of note, studies examining the differences between content-related versus process-related trait questionnaires regarding their predictive utility for depression and anxiety showed that an underlying RNT factor explained more variance in predicting anxiety and depression than disorder-specific cognitive content (i.e., rumination or worry) [64]. It remains to be tested whether the same applies to RNT assessed via EMA. Future studies should compare the two scales identified in the current study regarding the power to predict future symptom levels. As the two scales differ regarding their inclusion of content-related items, this will not only be informative from a purely psychometric but also from a theoretical point of view.

10. “In addition to examining whether an aggregated score of RNT across the EMA phase significantly predicts psychopathology, other dynamic measures can be taken into account, such as instability or stress-reactive RNT.” Flesh this idea out a bit more- I didn’t understand how the second half of the sentence related to the first.

RESPONSE: We have restructured this part of the discussion and have now added more information. The main message we wanted to get across is the idea that not only mean levels of daily RNT are important predictors of psychopathology, but that we can also exploit the advantages of EMA, i.e. taking into account the temporal dynamics of EMA variables (p. 22):

In addition to examining whether an aggregated score of RNT across the EMA phase significantly predicts psychopathology, other dynamic measures can be taken into account. For instance, one study has shown that EMA-assessed instability of rumination predicts depressive symptoms after six and 36 months over and above average levels of daily-life rumination [65]. Whereas the underlying causes for this finding remain to be investigated, the authors speculated that daily life stressors may be related to higher fluctuations in rumination, resulting in higher depression levels. Therefore, not only instability of RNT, but also stressors and the interaction of RNT with stress in daily life might be valuable targets for further investigation [66].

11. Supporting table 2 shows that FEEL, PROB, RPT, BUR has an RMSEA of .059. I am not expert in RMSEA but read that values below .06 indicated good fit. In which case, should this be noted in the body of your paper?

RESPONSE: Indeed, we found two models with acceptable fit, however, we decided to include the model with the best fit in further analyses. We, therefore, rephrased our results to clarify this (p. 14):

Two models were found to exhibit acceptable model fit: one model encompassing subjective burden (BUR) and uncontrollability of RNT (CTRL) (RMSEA = .047), another model encompassing subjective burden (BUR) and repetitiveness of RNT (RPT) (RMSEA = .059) in addition to both content items. We chose the former model including CTRL for further analyses, as it yielded the best model fit.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

1 Apr 2020

Assessing repetitive negative thinking in daily life: Development of an ecological momentary assessment paradigm

PONE-D-19-32809R1

Dear Dr. Rosenkranz,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements. This paper will make an excellent addition to the literature!

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Evan M Kleiman

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

7 Apr 2020

PONE-D-19-32809R1

Assessing repetitive negative thinking in daily life: Development of an ecological momentary assessment paradigm

Dear Dr. Rosenkranz:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Evan M Kleiman

Academic Editor

PLOS ONE