Identifying self-report measures of emotion regulation and evaluating their psychometric properties: a protocol for a systematic review.

INTRODUCTION: Successful emotion regulation (ER) is critical for psychological health. Disturbances in this ability are associated with several psychiatric disorders. There are several self-report questionnaires to assess ER. However, there are no studies synthesising the evidence on their psychometric properties. We aim to identify all available instruments addressing ER in adolescents or adults and to critically appraise, compare and summarise the quality of their psychometric properties. For this, we will use COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) criteria. METHOD AND ANALYSIS: The search process to identify eligible studies will be conducted in April 2021 including the ProQuest, PubMed, Scopus and Web of Science databases with no restriction in terms of publication date. Eligibility criteria include peer-reviewed research articles written in English or Spanish by means of patient-reported outcome measures focused on ER among participants of 13 years or older. We will assess the quality of measures according to the COSMIN Risk of Bias Checklist. The psychometric properties will be assessed by the COSMIN updated criteria for good measurement. The available evidence will be addressed by the Grading of Recommendations, Assessment, Development and Evaluations approach. Our findings will be synthesised independently for each measure, including information on their sample, theoretical model and psychometric properties when possible. ETHICS AND DISSEMINATION: Ethical approval is not required, as this study does not involve any participants or collection of primary data. Results are expected to be published in a peer-reviewed journal in the field of youth mental health and presented at relevant meetings and conferences. PROSPERO REGISTRATION NUMBER: CRD42021249498. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This protocol for systematic review has been prepared in accordance with the COSMIN (COnsensus-based Standards for the selection of health status Measurement INstruments) updated criteria.

The Cochrane Handbook for Systematic Reviews of Interventions was used to assist the design of this systematic review.

There are no recent syntheses of the evidence on psychometric properties of self-report measures for assessing emotion regulation in adolescent and adult populations.

This systematic review will be limited to published articles.

Searches will be limited to studies published in English and Spanish languages.

Introduction

Emotion regulation (ER) is the process where behaviours, competencies and strategies interact to modulate, inhibit, or intensify emotional experiences and expressions based on individual objectives, the circumstances they face and the development stage they are in.1 2 The ability to properly regulate our emotions is associated with greater subjective well-being3 and is essential for maintaining good mental health4 and successfully interacting with others.5 ER can be an important mediator to cope with adverse and stressful events.6 It has been observed that a lower ER capacity is related to the increased clinical symptoms,7–9 and that it constitutes a potential risk factor for the development and maintenance of psychopathologies,10 11 such as anxiety disorders, depression, personality borderline, post-traumatic stress disorder, psychotic spectrum disorders, and risk behaviours including substance use, self-injurious behaviours, and suicide attempts.6 12–23 In short, ER represents a complex psychological process associated with the development and manifestation of psychopathology,24 and it has special relevance to intervene on its course and outcomes.

Given the importance that ER has acquired in recent years, there has been a significant increase in its research. Different theoretical models have been developed that differ in the conceptualisation of ER, its strategies and the organisation of its components.25 26 Simultaneously, interventions have been developed to promote resilience and well-being27 as well as to treat a variety of psychiatric disorders with generally promising results.28 Likewise, different measures have been designed to assess specific components of ER. These measures include self-report questionnaires, behavioural observations, and peripheral and neural physiological measures. Because these measurements have different designs and structures, they are not always directly comparable.29 30

Among the theoretical models that underlie these instruments, it is possible to distinguish approaches that place greater emphasis on the adaptive aspect of ER.31 Gross’ modular model exemplifies this, as it focuses on the implementation of regulatory strategies to achieve the subject’s emotional regulation goals.32 Likewise, Thompson’s model highlights the importance of context as a factor influencing the subject’s regulatory goals in a particular situation.33 On the other hand, there are approaches focused on maladaptive aspects of ER that would account for alterations or difficulties in this process.34 Among these, one of the most representative models is that of Gratz and Roemer, who describe emotion dysregulation as experiencing difficulties in a series of dimensions of ER such as emotional awareness, acceptance and understanding, including the ability to regulate one’s own behaviour and impulsiveness in the face of negative emotions and accessing effective ER strategies.35

Despite the fact that literature suggests using multilevel measures to study this type of construct,36 self-report questionnaires are one of the most commonly used tools in ER research. While most of these measures have focused on the assessment of strategies for emotional suppression and cognitive reappraisal,30 others have been developed to assess aspects such as ER skills in addition to other regulatory strategies.26 37

Interest in measurement of these skills and strategies has risen rapidly in the last few years. As a result, some systematic reviews have been conducted on instruments for measuring ER. In general, these approaches have assessed the measurement of strategies in rather specific age groups and contexts, such as individuals with autism spectrum disorder,38 women with breast cancer,31 children and adolescents with intellectual disabilities,39 children’s ER in a school setting40 or evaluating specific strategies for coping with emotional states such as sadness.41 While these reviews have addressed the most relevant instruments and methods used to measure ER, the emphasis has been more on the measurement format (eg, self-report, interviews, behavioural observation) than on the instrument’s quality or psychometric properties. Moreover, most studies have reported only basic indicators such as internal consistency indices used to provide a broad assessment of the measure’s validity and reliability.41

The usage of self-report questionnaires to assess ER in adolescents has been highlighted as relevant and feasible.42 However, even if some initiatives have developed self-report measures to assess ER in children, they should be considered with caution because of the cognitive and affective developmental characteristics at this age that could interfere with the comprehension of the questions about emotions and their management.43 Furthermore, there is evidence that the cognitive processes and mechanisms required for effective implementation of ER strategies develop during adolescence.44 45 Considering these findings, our study focuses on studies involving participants aged 13 years or older in order to include adolescence as an age group based on the classification by the WHO,46 while also reducing potential developmental biases in the comprehension of items requiring a higher level of abstract thinking.47

Through our systematic review, we seek to identify the existing self-report instruments for the measurement of ER, determining their psychometric properties and overall quality. With this, our goal is to identify which instruments have the best properties and prove to be the most valid and reliable for measuring strategies for or difficulties in ER skills. Given the advantages of self-report instruments (eg, quick to administer, easy to score and the suitability for being used as part of a broad survey),48 we focused on these measures. This fits with the current recognised need to improve the early detection of mental health difficulties by addressing a broad range of symptoms and underlying transdiagnostic mechanisms.49 50

The following elements form the basis of the current systematic review: (1) ER is currently considered as a vital process for understanding psychopathology and developing effective interventions; (2) it is critical to be clear regarding measurement of ER, its underlying theoretical models, validity evidence and the samples used51; (3) there have been no previous reviews that synthesise the validity evidence for self-report measures to assess ER. Therefore, the current systematic review will summarise the evidence on the psychometric properties of self-report questionnaires used to measure ER, particularly among people over the age of 13 years. The objectives include:

To identify all available instruments to assess ER in adolescents and adults.

To critically assess, compare and synthesise the measurement properties of the identified instruments, based on the criteria of the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN).

Method

The following protocol for systematic review was developed in accordance with the COSMIN initiative’s criteria.52 The protocol’s details have been registered in the International Prospective Register of Systematic Reviews. In the event of modifications to the protocol, these will be informed in the publication of the systematic review. The planned start date for the systematic review was on 1 April 2021 and the end date was planned on 30 December 2021. The procedure is summarised in figure 1.

Figure 1

Flow chart for the systematic review process under the COSMIN guidelines. COSMIN, COnsensus-based Standards for the selection of health Measurement INstruments; GRADE, Grading of Recommendations, Assessment, Development and Evaluations; PROM, patient-reported outcome measure.

Search strategy

For the database search, we will combine the following search terms and related free-text grouped in blocks, among which contain “emotion regulation”, “emotion dysregulation”, “instruments”, “scale”, “questionnaire”, “test”, “re-test”, “validation” and “psychometrics properties”; these terms can be present in both the title and the abstract. The specific search strategies for each database are included in the online supplemental appendix 1.

Information sources

An exhaustive literature search will be carried out in electronic databases such as Scopus, Web of Science, PubMed and ProQuest (MEDLINE). The search began in April 2021. In order to detect studies that may have been omitted during the search (and that may contribute to the systematic review), the search will include a manual review of the references of the included articles.

Eligibility criteria

Inclusion criteria

The criteria defined for the inclusion of the studies include: (1) research articles published in peer-reviewed journals, without a date range for their indexing in the previously described databases; (2) research articles addressing instruments for measuring emotion regulation or dysregulation strategies and their psychometric characteristics; (3) research articles written in either English or Spanish.

Exclusion criteria

The criteria defined for the exclusion of the studies include: (1) being a theoretical or non-empirical review, case studies, thesis, conference abstract, systematic review or meta-analysis; (2) not presenting psychometric properties of the instruments; (3) the average age of the participants being less than 13 years old; (4) using an ER measure other than a self-report; (5) ER measurement not being the main focus of the study. Measures of internal consistency, reliability, measurement error, content validity, structural validity, hypothesis testing for construct validity, cross-cultural validity, criterion validity and responsiveness will be understood as psychometric properties.

Study records

Data management

All records will be stored in comma-separated values files. Microsoft Excel functions along with manual checks will be performed to identify potential duplicates.

Study selection process

In the first step, two reviewers (CV-H and KC) with experience in database management will conduct and consolidate the search results. After identifying the records in the databases and eliminating the duplicates, the study selection process will be carried out. During the second step, the collected articles will be evaluated on their relevance to the particular review by checking their titles and abstracts. If the relevance of an article cannot be determined with this information, the full text will be reviewed to determine its eligibility, applying at all times the inclusion and exclusion criteria previously described.

This assessment will be carried out independently by both reviewers. A third reviewer (DN or JLU) will intervene in case of any disagreements regarding the inclusion or exclusion of an article. For the study selection, criteria evaluation and data extraction, a calibration phase will be considered. In this phase, an initial number of studies will be randomly selected and evaluated iteratively until agreement is reached among the reviewers. This process is carried out to guarantee the homogeneity of criteria in the review process.

Data extraction process

In the third step, two reviewers (CV-H and KC) will independently perform data extraction from included studies, assessment of risk of bias and assessment of the measurement properties’ quality based on COSMIN guidelines for systematic reviews of patient-reported outcome measures (PROMs). To characterise the studies included in the systematic review, a narrative synthesis will be performed accompanied by comparative tables. The assessment of the studies and data extraction will be carried out based on the COSMIN Risk of Bias Checklist and the updated COSMIN criteria for the evaluation of measurement properties.53 54

Risk of bias assessment

The COSMIN Risk of Bias Checklist will be adopted to assess the risk of bias. This instrument has been developed exclusively for the systematic review of PROMs. The checklist comprised of 10 main categories that will be used to evaluate: (1) PROM development; (2) content validity; (3) structural validity; (4) internal consistency; (5) cross-cultural validity; (6) reliability; (7) measurement error; (8) criterion validity; (9) hypothesis testing for construct validity; (10) responsiveness.54 These aspects will be evaluated by both reviewers independently and discussed until a consensus on their quality is reached. Each study will be classified as one of the following: ‘very good’ (V), ‘adequate’ (A), ‘doubtful’ (D), ‘indeterminate’ (I) or ‘not applicable’ (N).

Measurement properties assessment

The updated COSMIN criteria for the evaluation of measurement properties will be used to evaluate the psychometric properties of each instrument included in the study. The criteria can assess aspects such as structural validity, measures of internal consistency, reliability, measurement error, construct validity, cross-cultural validity, criterion validity and responsiveness. These aspects are evaluated by both reviewers who reach an agreement regarding whether their quality is sufficient (+), insufficient (−) or indeterminate (?). The criteria are summarised in table 1.

Table 1

COSMIN updated criteria for good measurement properties

Measurement property	Rating	Criteria
Structural validity	+	Classical Test Theory (CTT)Confirmatory Factor Analysis (CFA): Comparative Fit Index (CFI) or Tucker-Lewis Index (TLI) or comparable measure >0.95 or Root Mean Square Error of Approximation (RMSEA) <0.06 or Standardized Root Mean Square Residual (SRMR) <0.08
		IRT (Item Response Theory)/RaschNo violation of unidimensionality: CFA: CFI or TLI or comparable measure >0.95 OR RMSEA <0.06 OR SRMR <0.08ANDNo violation of local independence: residual correlations among the items after controlling for the dominant factor <0.20 OR Q3’s <0.37ANDNo violation of monotonicity: adequate-looking graphs OR item scalability >30ANDAdequate model fit: IRT χ2>0.01; Rasch: infit and outfit mean squares ≥0.5 and ≤15 OR Z-standardised values >−2 and <2
	?	CTTNot all information for ‘+’ reported
		IRT/RaschModel fit not reported
	–	Criteria for ‘+’ not met.
Internal consistency	+	At least low evidence for sufficient structural validity AND Cronbach’s alpha(s) ≥0.70 for each unidimensional scale or subscale
	?	Criteria for ‘At least low evidence for sufficient structural validity’ not met
	–	Criteria for ‘+’ not met
Reliability	+	Intraclass correlation coefficient (ICC) or weighted kappa ≥0.70
	?	ICC or weighted kappa not reported
	–	ICC or weighted kappa <0.70
Measurement error	+	Smallest Detectable Change (SDC) or Limit of Agreement (LoA) <Minimal Important Change (MIC)
	?	MIC not defined
	–	SDC or LoA >MIC
Hypotheses testing for construct validity	+	The result is in accordance with the hypothesis
	?	No hypothesis defined (by the review team)
	–	The result is not in accordance with the hypothesis
Cross-cultural validity/measurement invariance	+	No important differences found between group factors in multiple group factor analysis OR no important differential item functioning (DIF) for group factors (McFadden’s R<0.02)
	?	No multiple group factor analysis OR DIF analysis performed
	–	Important differences found between group factors in multiple group factor analysis OR DIF was found
Criterion validity	+	Correlation with gold standard ≥0.70 OR area under the curve (AUC) ≥0.70
	?	Not all information for ‘+’ reported
	–	Correlation with gold standard <0.70 OR AUC <0.70
Responsiveness	+	The result is in accordance with the hypothesis OR AUC ≥0.70
	?	No hypothesis defined (by the review team)
	–	The result is not in accordance with the hypothesis OR AUC <0.70

+=sufficient; ?=indeterminate; −=insufficient.

COSMIN, COnsensus-based Standards for the selection of health Measurement INstruments.

Data synthesis

To characterise the studies included in the systematic review, a narrative synthesis with comparative tables will be performed. In addition, each study will receive a critical evaluation. The psychometric properties of the instruments included will be evaluated through the updated COSMIN criteria for the evaluation of measurement properties. The results will be synthesised through a general evaluation of the PROM with the same criteria. This assessment will consider the article in which the instrument was developed and its subsequent validations. Short or substantially modified versions of an instrument will be evaluated as independent from the original version.

The quality of the available evidence will be assessed through a modified Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach, in which both reviewers reach a consensus on whether the available evidence of each assessed aspect is ‘high’, ‘moderate’, ‘low’ or ‘very low’.55 The results of this evaluation will be included in conjunction with the previously described results.

The evidence for each measurement property will be summarised for each measurement instrument, and the overall result will be determined based on the criteria of good measurement properties. The quality of the evidence will be graded according to a modified GRADE approach (‘high’, ‘moderate’, ‘low’, ‘very low’ evidence). This general quality synthesis will be used to determine which measures of ER are more robust for the sample population in which they are used.

Patient and public involvement

This research does not involve any patients or public members. No patients participated in the design of the protocol.

Ethics and dissemination

This study was based on previously published data and did not directly involve any human participants. Therefore, it is exempt from ethical review. Results are expected to be published in a peer-reviewed journal in the field of adolescent and/or youth mental health.