The Campbell Collaboration's systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards.

BACKGROUND: Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration's mandatory methodological standards. ISSUES: In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review.
CONCLUSIONS: Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Many published systematic reviews are poorly conducted, and many reviews do not follow widely accepted Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) standards (http://www.prisma-statement.org), although the quality of reporting of biomedical reviews has improved over time [1]. Campbell Collaboration and Cochrane reviews are expected to meet more rigorous standards. The methodological quality and reporting characteristics of Campbell Collaboration reviews have improved over time, but only 17% of a sample of 96 Campbell reviews were assessed as high quality [2].

Campbell Collaboration systematic reviews aim to provide rigorous, transparent, and unbiased assessments of research evidence, so that readers can have confidence in their methods and conclusions. Their guidelines state: “Every Campbell review is required to have clear criteria for eligible research, an explicit and comprehensive search strategy, systematic and replicable coding and analysis of the key features and findings of the studies reviewed, and an integrative summary of those findings” ([3] p. 5). In 2014, the Campbell Collaboration established an explicit set of Methodological Expectations for Campbell Collaboration Intervention Reviews to guide the conduct and reporting of its reviews [2, 4, 5]. Each MECCIR standard is identified by a number, following C for conduct or R for reporting.

Initially, the updated Campbell Collaboration review of school-based anti-bullying interventions [6] caught our attention because the vast majority of its included studies were rated as having low risk of selective outcome reporting (SOR) bias. Under-reporting and selective reporting of outcomes are common [7-11] and preregistration of behavioral intervention trials is uncommon [12-15], so we were curious to know how this review determined whether “outcomes reported in an evaluation study differ from the outcomes of interest proposed originally” ([6] p. 56). Upon further examination, we found that this review did not appear to meet many mandatory MECCIR standards. Five versions of this review were published outside of the Campbell Collaboration [16-20]. Such wide dissemination of results has the potential to influence educational policy and practice, so it is important to understand the review’s methods and conclusions.

Below we raise concerns about methodological qualities of this review and the confidence that readers can place in its results. We describe the Campbell Collaboration’s response to these concerns, and document a difference between their published standards and publication decisions.

Methodological issues

Unexplained deviations from the protocol

In our assessment, the new Campbell Collaboration school-based anti-bullying interventions review does not fully “explain and justify any changes from the protocol” (mandatory MECCIR standard R106). Post hoc changes were made in study inclusion criteria, and 13 previously-included studies were excluded as a result [6]. It is not clear why “other quasi-experimental designs” were excluded from the review but “age cohort designs” were retained, as the latter were likely to have different threats to internal validity [21], including history and testing effects, and other uncontrolled differences between groups [6]. To our knowledge, history and testing effects were not assessed in this review. An earlier report indicated that the largest effect sizes were found in age cohort designs [19]. It is possible that post hoc changes in study inclusion criteria affected overall results, but no sensitivity analyses were provided to assess potential impacts of departures from the protocol on overall results (as per MECCIR C13).

The review’s risk of bias (ROB) assessments and moderator analyses also deviated from plans described in the protocol [22] but these changes were not explained (as required by MECCIR R106). There were no plans for ROB assessment in the protocol (original plans for extraction of data on study qualities focused only on overall study designs and attrition). Plans for moderator analysis were not specific, indicating that meta-regression would be used to “investigate independent influences of program components, methodological quality, features of participants, and design features” ([22] p. 12).

Inadequate documentation of search strategies

We found that search strategies were not reported in sufficient detail for replication (MECCIR C36, R34, R35, R36, R38, R39). Systematic searches were completed in December 2016 [6], more than 4 years prior to publication. Exact search strings, dates, and limits were not provided for specific databases. It is not clear what sources other than ProQuest Dissertations and Theses Solutions were used to search for grey literature [6]. Five studies were added “after searches” were completed ([6] pp. 6, 17), and a sixth study [23] appears without an explanation.

Inconsistent reports on the number of included studies

The MECCIR requirements state that reviews should fully account for the status of studies (MECCIR C40, C41, C42) and comment on the potential impact of included studies without useable data (MECCIR R89). We found that these steps were not taken in the school-based anti-bullying interventions review.

At one point, the review states that 88 newly identified studies were included ([6] pp. 17, 51); elsewhere it reports that 79 new studies were included ([6] p. 52); but results are shown for only 74 studies ([6] p. 21), with no explanation for missing studies. Portions of the review indicate that 45 RCTs were included ([6] pp. 2, 21), but results are shown for only 41 RCTs ([6] p. 21); again, with no explanation for missing studies.

The review states that total of 141 (old and new) studies were included, then 13 studies were dropped due to post hoc changes in inclusion criteria, bringing the revised total of included studies to 128 ([6] p. 52). In all, 41 studies were excluded for reasons related to study design or incomplete data (in conflict with MECCIR C40) and 100 studies were included in the meta-analysis. References are provided for 116 included studies.

The status of one study [24] is unclear. The reference for this study appears on a list of excluded studies ([6] p. 95) and a similar citation appears in a table of excluded studies ([6] p. 16); but this study is listed as an included study in three places ([6] pp. 26, 75, 101).

Undocumented ratings of risks of bias

Campbell Collaboration reviews must “present a ‘Risk of Bias’ and/or ‘Study Quality’ table for each included study, with judgments about risks of bias, and explicit supports for these judgments” (MECCIR R72). Campbell reviews are expected to “justify categorical risk of bias/study quality judgments (e.g., high, low, and unclear) with information [taken] directly from the study” (MECCIR C53). But the Campbell school-based anti-bullying interventions review does not provide support for judgments about risk of bias. The review only listed categorical ratings (L for low risk, U for unclear risk, and H for high risk) and provided an overall (study-level) risk of bias score for 90 studies ([6] Appendix B).

Assessment of the inter-rater reliability of risk of bias (ROB) ratings is considered best practice (MECCIR C45, C52). But there was no systematic double coding in this review and there is no information on the reliability of any data extraction or coding tasks ([6] p. 6).

Assessment of selective outcome reporting (SOR) bias

The review provided SOR bias ratings for only a subsample of included studies: SOR bias ratings are reported for 89 studies on pages 55 and 56 and for 90 studies in Appendix B, but not for all 100 studies included in the meta-analysis or for any of the 41 studies that met initial inclusion criteria but were not included in meta-analysis. Almost all (94%) of the studies that were rated were characterized as having low risk of SOR bias; only two studies were rated as high risk of SOR, and three were rated unclear.

The review did not use an established ROB tool for assessment of SOR bias. According to the published review, “SOR occurs when the outcomes reported in an evaluation study differ from the outcomes of interest proposed originally. For example, if a trial protocol proposed different outcomes than those actually reported in the publication of the trial results” ([6] p. 56). The review defined two levels of SOR bias: a rating of low risk of SOR bias was assigned when “Outcomes proposed are outcomes that are reported” and high risk of SOR bias was identified when “Outcomes proposed are not the outcomes that are reported” ([6] p. 19). The review does not indicate whether protocols for included studies were retrieved or how it was determined which outcomes were “proposed” for each study if prospectively registered protocols were not available. Further, the review provides no documentation of sources consulted or explanations for SOR bias ratings for each study, as required by Campbell’s mandatory MECCIR standard R72.

In the absence of additional support for assessments of SOR bias, we attempted to replicate these ratings, using the review’s criteria and the references it provided. We assumed that randomized controlled trials (RCTs) were more likely than other study designs to have been prospectively registered or have publicly available protocols, so we retrieved documents cited in relation to 42 unique RCTs listed in Tables 9 and 10 of the review [6]. Table 1 (below) shows the SOR ratings the review provided for these 42 RCTs, along with results of our attempt to verify these ratings for the 41 study reports we were able to locate (we could not retrieve one report from a German journal).

Table 1

RCTs focused on school-bullying perpetration and/or victimization

Study a	SOR bias rating in the reviewb	RCT report mentions registry or protocolc	Our rating using criteria stated in the reviewd
1. Baldry and Farrington (2004) [25]	Low	No	Unclear
2. Beran and Shapiro (2005) [26]	Low	No	Unclear
3. Berry and Hunt (2009) [27]	Low	No	Unclear
4. Bonell et al. (2015) [28]	Low	Yes	Low
5. Boulton and Flemington (1996) [29]	Low	No	Unclear
6. Brown et al. (2011) [30]	Low	No	Unclear
7. Chaux et al. (2016) [31]	Low	No	Unclear
8. Cissner and Ayoub (2014) [32]	Low	No	Unclear
9. Connolly et al. (2015) [33]	Low	No	Unclear
10. Cross et al. (2011) [34]	Low	No	Unclear
11. DeRosier and Marcus (2005) [35]	Low	No	Unclear
12. Domino (2013) [36]	Low	No	Unclear
13. Espelage et al. (2015) [37]	Low	No	Unclear
14. Fekkes et al. (2006) [38]	Low	No	Unclear
15. Fekkes et al. (2016) [39]	Low	No	Unclear
16. Fonagy et al. (2009) [40]	Low	No	Unclear
17. Frey et al. (2005) [41]	Low	No	Unclear
18. Garaigordobil and Martínez-Valderrey (2015) [42]	Low	No	Unclear
19. Holen et al. (2013) [43]	Low	No	Unclear
20. Hunt (2007) [44]	Low	No	Unclear/high
21. Jenson et al. (2013) [45]	Low	No	Unclear
22. Ju et al. (2009) [46]	Low	No	Unclear/high
23. Kaljee et al. (2017) [47]	Unclear	No	Unclear
24. Kärnä et al. (2011b) [48]	Low	No	Unclear
25. Kärnä et al. (2013) [49]	Low	No	Unclear
26. Knowler and Frederickson (2013) [50]	Low	No	Unclear
27. Krueger (2010) [51]	Low	No	Unclear
28. Li et al. (2011) [52]	Low	No	Unclear
29. McLaughlin (2009) [53]	Low	No	Unclear
30. Meyer and Lesch (2000) [54]	Low	No	Unclear
31. Nocentini and Menesini (2016) [55]	Low	No	Unclear
32. Ostrov et al. (2015) [56]	Low	No	Unclear/high
33. Polanin (2015) [57]	Low	No	Unclear
34. Rosenbluth et al. (2004) [58]	Low	No	Unclear
35. Spröber et al. (2006) [59]	Low	Unknowne	Unknown
36. Stallard et al. (2013) [60]	Low	Yes	High
37. Topper (2011) [61]	Unclear	No	Unclear
38. Trip et al. (2015) [62]	Low	No	Unclear
39. Tsiantis et al. (2013) [63]	Low	No	Unclear
40. Waasdorp et al. (2012) [64]	High	No	Unclear
41. Wölfer and Scheithauer (2014) [24]	Low	No	Unclear/high
42. Yanagida et al. (2019) [23]	Low	No	Unclear
Summary	39 Low, 1 High, 2 Unclear	2 Yes, 39 No, 1 Unknown	1 Low 1 High, 4 Unclear/high, 35 Unclear, 1 Unknown

aCited in the review [6]. Bold font = RCTs in both Tables 9 and 10 (k = 26); normal font = RCTs in Table 9 (perpetration) only (k = 9); Italic font = in Table 10 (victimization) only (k = 7)

bFrom Appendix B of the review [6]

cEach document was electronically searched for the words “Registry”, “Registered”, “Registration”, and “Protocol”. “No” = this search did not yield a reference to a registry or study protocol; “Yes” = this search did yield a reference to a registry or publicly available study protocol

dCriteria provided in the review ([6] p. 19)

eThis study is in a German journal. It could not be located through Inter-library Loan; therefore, we could not rate the risk of SOR bias for this study

Only two of the 41 reports on RCTs make reference to trial registration or a public study protocol. The details of these studies are important, as they clearly illustrate issues encountered in assessing risk of SOR bias.

An RCT by Stallard and colleagues [60] was prospectively registered in 2007 [65]. The intervention tested in this trial was not an anti-bullying program (it aimed to prevent depression) and the trial registration record does not mention any intended outcomes related to bullying [65]. Enrollment began in 2008 and data collection began in 2009 [65]. A second protocol for this study, published retrospectively, mentions bullying as one of several secondary outcomes, but does not indicate how bullying outcomes were measured [66]. A third “protocol” for this study appears as an appendix to the 2013 study report and it states that, “The two global items [of the Olweus Bully/Victim Questionnaire] assessing the frequency of self-reported bullying and being the victim of bullying will be used” ([60] p. 105). However, Stallard and colleagues reported results for only one of these two bullying outcomes (perpetration, not victimization) [60]. This study was rated as low risk of SOR in the review, although it meets the review’s criteria for high risk, because bullying outcomes were not mentioned in the initial proposal.

A trial reported by Bonell and colleagues [28] was prospectively registered in 2011 [67], listing aggressive behaviors and bullying as primary outcomes to be measured by the Gatehouse Bullying Scale (12 items) and the Ayan Aba Youth Project subscale on aggressive behavior (4 items). Results are reported for all 16 items at baseline and follow-up [28]. The study was rated as low risk of SOR bias in the review [6], which is justified based on a comparison of the trial registry record and the study report.

The remaining 39 reports on RCTs included no references to trial registration or public protocols (see Table 1). The review rated 36 of these trials as low risk of SOR bias, two as unclear, and one as high risk. Using the review’s stated criteria, we rated all 39 of these RCTs as having unclear risk of SOR bias; because no prospectively registered protocol was cited in these trials, there was no way to determine which outcomes were initially proposed.

A closer look at the trial reports raised additional questions about the review’s assessment of SOR bias. Two studies collected data on the Olweus bully/victim questionnaire but reported results for only one of the two outcomes assessed with this instrument (one study reported perpetration only [24], another reported victimization only [46]). A third study obtained outcome measures on four types of bullying (proactive physical bullying, reactive physical bullying, proactive relational bullying, and reactive relational bullying), but collapsed proactive and reactive measures in the reported analysis ([56] p. 450). A fourth trial obtained data on two of the Attitude to Bullying subscales but included only one of these subscales in the published analysis ([44] p. 23). We coded these four studies as having Unclear/High risks of SOR. The review rated a fifth study [64] as having a high risk of SOR bias; given the lack of a study protocol or trial registration record, we rated the risk of SOR in this study as unclear.

In summary, using the review’s criteria for SOR bias, we rated 35 of 41 trials as having unclear risk, four as unclear/high risk, one trial as high risk, and one trial as low risk (see Table 1). As shown in Table 2, our ratings agree with those of the review on only 3 of 41 trials (proportion of agreement= 7%, Cohen’s kappa = 0.003), a very low level of agreement.

Table 2

Reliability of ratings of SOR bias for 41 RCTsa

	Our SOR ratings based on criteria stated in the review ([6] p. 19)
SOR ratings in the review ([6] Appendix B)	Low risk	Unclear or unclear/high	High risk
Low risk	1	36	1
Unclear	0	2	0
High risk	0	1	0

aFor 41 trials with two sets of ratings: proportion of agreement = 7% (3/41); Cohen’s kappa = 0.003

The review rated all but two (45) non-RCTs as having low risk of SOR bias. We did not attempt to verify these ratings because we did not expect to find protocols for these studies.

Selective reporting of outcomes is a pervasive problem in evaluations of interventions in the social and health sciences [8, 9, 11], and SOR bias remains a clear threat to the validity of systematic reviews [7, 68]. We found that the methods used in this review to assess SOR are not transparent, not replicable, and appear to systematically underestimate risk of SOR bias in the included studies.

Unreliable assessments of risk of publication bias

To assess the risk of publication bias, the review relied on (a) visual inspection of funnel plots and (b) trim and fill analysis. Empirical studies show that visual assessment of funnel plot asymmetry is unreliable [69, 70]. It is not clear why reviewers did not use Egger’s test or another statistical test of funnel plot asymmetry. Trim and fill analysis is not reliable in the presence of between-study heterogeneity [71, 72], and substantial heterogeneity is apparent in this review ([6] p. 76). Results of trim and fill analysis depend heavily on which estimators are used [73], but estimators were not specified in the review. In sum, the review does not provide convincing evidence for its conclusion that publication bias was unlikely or “not present” in its meta-analyses ([6] pp. 2, 74, 76).

Use of a composite scale that conflates distinct risks of bias

The review states, “Scores on each of the risk of bias items were summed to estimate a total risk of bias score. This continuous variable was then used to examine the relationship between intervention effectiveness and risk of bias in meta-regression models” ([6] p. 76). This is at odds with a mandatory MECCIR standard (C51), which states that “Campbell reviews should not use composite scales, indices, or other measures that conflate multiple measures of risk of bias/study quality into a single score (e.g., using an average scale that combines measures of allocation concealment, attrition, and baseline equivalence measures). These composite quality scales can be misleading and should not be used in a Campbell Collaboration review. Instead, any risk of bias/study quality coding should isolate unique measures of quality (e.g., separate measures for allocation concealment, attrition, spillover, selective outcome reporting, selective analysis reporting, and baseline equivalence)” ([4] p. 16).

Considering the strength of the evidence in interpreting results and drawing conclusions

The review rated 30% to 40% of (k = 89 to 91) included studies as having high risks of bias on: allocation sequence, allocation concealment, contamination, and conflicts of interest ([6] pp. 55–56, Appendix B). It reported a mismatch between units of allocation and units of analysis in most studies, noting that this was “a threat” to the findings ([6] p. 86). Studies rated high risk of conflict of interest (COI) had significantly larger effect sizes than studies rated low risk of COI ([6] p. 84). Yet there was no discussion of issues of risk of bias (or study quality) in the review’s abstract (mandatory MECCIR items R11 and R12) or in the discussion of limitations of the review (MECCIR mandatory item R100). We think these issues should have been presented as caveats for readers to consider when evaluating the review’s conclusions that school-based anti-bullying programs “are effective” and their “effect sizes are modest.”

Conclusions

The new (updated) Campbell Collaboration systematic review of school-based anti-bullying interventions [6] does not appear to meet many of the Campbell Collaboration’s mandatory MECCIR standards (e.g., C13, C20, C22, C36, C51, R11, R12, R34, R35, R36, R38, R72, R89, R100, R106). These standards were created to support the conduct and reporting of systematic reviews. In our assessment, the review’s deviations from mandatory MECCIR standards mean that its searches are not replicable, inclusion decisions are not transparent (numbers of included studies fluctuation for reasons that are unclear), and bias assessments are not supported with evidence. Most striking, our assessment of SOR bias showed very little agreement with the review’s SOR bias ratings, even though the same criteria were used in both assessments. We believe that this review underestimates risks of SOR and publication bias, which may lead readers to think that the evidence base for this review is more complete and more trustworthy than it really is. Further, the review does not fully consider issues related to the strength of the evidence and risks of bias when presenting its conclusions. We believe this raises questions about the confidence readers can place in this review.

Our assessment also raises concerns about the editorial process that led to publication of this review. The Campbell Collaboration MECCIR reporting standards state that “a new review will not be published if [a mandatory] standard is not met” ([5] p. 3). Give our assessment that these standards were not met, it is not clear to us why this review was published in Campbell Systematic Reviews.

Clear standards for the conduct and reporting of systematic reviews provide important guidance for reviewers and editors. Readers should be able to assume that established guidelines were followed, and mandatory requirements were met. This is especially important if readers are using systematic reviews to guide their decisions as to which intervention programs to implement. In a world of diminishing resources for social interventions, there are opportunity costs associated with selecting poorly evaluated interventions that produce unreliable, biased, or false positive results. Systematic reviews are intended to differentiate weak studies from rigorous evaluations that produce valid results, using thorough assessments of common sources of bias, such as SOR. Our confidence in this review was reduced by the lack of transparency both within the review and in the editorial process.

With regard to the latter, we should note that Campbell Collaboration editors declined to publish our comments on this review (they would only consider publishing a brief comment on one issue: assessment of SOR bias). They also declined to publish a statement alerting readers to the fact that concerns had been raised about whether the updated school-based anti-bullying interventions review met many of the mandatory MECCIR standards.

We think that greater transparency about the application of published standards (including questions about whether a specific review has met these standards) and about editorial and publication decisions is needed to instill readers’ confidence in these processes and improve the quality of future systematic reviews. If mandatory MECCIR standards are not followed in published Campbell reviews, then there is a real gap between the Campbell Collaboration’s public criteria and its editorial and publication decisions. This gap is not transparent and the de facto standards for Campbell Collaboration systematic reviews are not clear to us. Lack of transparency and erosion of published standards may diminish the rigor of research reviews and undermine public confidence in them.

For us this raises questions about whether the Campbell Collaboration is living up to the legacy of Donald T. Campbell, the US social scientist for whom the Collaboration was named. In 1986, Campbell wrote, “Science requires a disputatious community of ‘truth seekers’” ([74] p. 35). He added, “The norms of science are explicitly anti-authoritarian [and] antitraditional…. The community of scientists is to stay together in focused disputation, attending to each other’s arguments and illustrations, mutually monitoring and ‘keeping each other honest’…” ([74] p. 35). Following Donald T. Campbell, we believe there is ongoing need for more public debate about the methods and conclusions of systematic reviews and all forms of empirical research. Further, editorial and publication decisions should be more transparent and open to public debate. Science cannot flourish otherwise.