Dos and Don'ts in Designing School-Based Awareness Programs for Suicide Prevention.

Background: Despite the promising evidence for the effectiveness of school-based awareness programs in decreasing the rates of suicidal thoughts and suicide attempts in young people, no guidelines on the targets and methods of safe and effective awareness programs exist. Aims: This study intends to distill recommendations for school-based suicide awareness and prevention programs from experts. Method: A three-stage Delphi survey was administered to an expert panel between November 2018 and March 2019. A total of 214 items obtained from open-ended questions and the literature were rated in two rounds. Consensus and stability were used as assessment criteria.
Results: The panel consisted of 19 participants in the first and 13 in the third stage. Recommended targets included the reduction of suicide attempts, the enhancement of help-seeking and peer support, as well as the promotion of mental health literacy and life skills. Program evaluation, facilitating access to healthcare, and long-term action plans across multiple levels were among the best strategies for the prevention of adverse effects. Limitations: The study is based on opinions of a rather small number of experts.
Conclusion: The promotion of help-seeking and peer support as well as facilitating access to mental health-care utilities appear pivotal for the success of school-based awareness programs.

The incidence of suicidal thoughts and suicide attempts increases steeply from the age of 12 (Cha et al., 2018). Although the suicide rate in young people is about half the suicide rate in older adults (Bertolote & Fleischmann, 2002), suicide is nevertheless the third most common cause of death for adolescents aged 15–19 years worldwide (World Health Organization [WHO], 2019). School-based prevention programs are often considered among the key strategies of prevention for adolescents (Surgenor et al., 2016).

School-based suicide prevention programs can be categorized as awareness/education programs, gatekeeper education, and screening programs (Robinson et al., 2013). Awareness programs are universal prevention strategies that aim to raise awareness of suicidality and provide education regarding recognition and help-seeking. Gatekeeper education and screening programs aim at identifying young people who may be suicidal through trained individuals or screening instruments, and referring those in need to appropriate services (Gould et al., 2003; Robinson et al., 2013). Wasserman et al. (2015) demonstrated in a multi-centered, randomized controlled trial (RCT) that the Youth Aware of Mental Health program led to reduced suicide thoughts and attempts after 12 months compared with a minimal intervention. A gatekeeper education program for school personnel and a screening program had no such effect. Similarly, the authors of the program Signs of Suicide reported a reduction of suicide attempts in several RCTs (Katz et al., 2013). In a recent systematic review of reviews, Platt and Niederkrotenthaler (2020) reported that awareness programs seem to be the type of school-based prevention with the highest level of evidence in suicide prevention. However, further high-quality evidence is required to prove the effectivity and usefulness of these programs. This is especially important in light of the substantial amount of evidence currently supporting the use of other universal prevention strategies, such as restrictions on common methods of suicide.

Several studies have reported unexpected negative effects of awareness programs, such as higher rates of negative attitudes toward help-seeking (Kuiper et al., 2019). Research on awareness programs has been further hampered by the lack of consensus regarding the safest and most effective methods, the need for large sample sizes and follow-up periods, as well as ethical concerns (Kutcher et al., 2017; Nordentoft, 2011). Evidence-based guidelines could therefore help to inform the quality and safety of future programs. Surgenor et al. (2016) developed 10 general recommendations for school-based suicide prevention programs based on a scoping review. In this study, we used the expert Delphi survey method to formulate recommendations focusing on school-based awareness programs for suicide prevention, more specifically targeting: (1) which contents should be included and what would constitute favorable outcomes, (2) what precautions could prevent adverse effects, and (3) what formats and techniques would be particularly useful?

Method

A Delphi survey is a multi-stage survey incorporating multiple rounds of questionnaires administered consecutively to a panel of experts (Jones & Hunter, 1995; Jorm, 2015). After each round, participants are provided with individualized feedback containing the overall response of the panel. Experts are then asked to re-evaluate their initial responses. The Delphi process is repeated until a pre-arranged criterion is achieved. The most common criteria are consensus among participants or stable responses between subsequent rounds (von der Gracht, 2012). The Delphi survey in this report was conducted from November 2018 to March 2019 (Round 1: November 20 to December 6, 2018; Round 2: December 21, 2018 to January 9, 2019, Round 3: February 14 to March 3, 2019).

Formation of the Delphi Panel

We recruited international researchers in the fields of suicidality and suicide prevention. We aimed to reach a sample size of seven or more, as this number has been suggested to be the minimum required for stable group decisions (Häder, 2014). Researchers were considered to be experts if they contributed significantly to research on suicidality or had specific knowledge in awareness programs for suicide prevention in schools, which means:

They were the first authors of two or more research papers on suicidality or suicide prevention AND

Had a Web of Science H-index > 10 OR

Led at least one research project on a school-based awareness program.

To identify participants, one of the authors (K.He) conducted a literature search on GoogleScholar, Hogrefe, eLibrary, and PsychInfo with the search terms: [“suicide*” OR “suicide prevention”] AND [“youth” OR “adolescent*” OR “school”] AND [“awareness” OR “curriculum” OR “program”]. The same author screened the results for relevant articles and generated an initial list of 76 researchers active in suicide research. Overall, 63 researchers met the inclusion criteria and were invited via e-mail to participate in the survey. Asking the candidates to suggest further experts in the field did not produce any further names.

Qualitative Survey (Round 1)

In the first round of the survey, the expert panel received open-ended questions generated by the research team (Table 1). Participants were encouraged to record up to five answers per question. The panel members completed the questionnaire via the online platform SoSci Survey (Leiner, 2016).

Table 1

Questions and rating criteria

Theme	Question (qualitative survey)	Criteria and rating scales (quantitative survey)
Note. The questions in this table are a short version of the original ones. aScale: 1 = very detrimental to 5 = very effective. bScale: 1 = not at all to 4 = to a great extent. cScale: 1 = strongly disagree to 4 = strongly agree. dScale: 1 = I do not recommend to do this to 4 = very important. eScale: 1 = not necessary to 5 = very useful. fScale: 1 = detrimental to 5 = very important; ranking of profession from the most to the last indicated.
1. Content	Essential topics you recommend to include in an awareness program?	Effectiveness for: (a) mental health promotion; (b) suicide preventiona
2. Favorable outcomes	Behavioral, emotional, and psychological effects that should be achieved?	(a) Feasibility in a short interventionb; (b) effectiveness in reducing suicidality in the long termb; (c) importance for awareness programs (five most important outcomes)
3. Adverse effects	Precautions to prevent unanticipated consequences?	(a) Agreement with general statements on the safety of awareness programsc; (b) importance of precautions for the prevention of adverse effectsd
4. Formats and techniques	Recommended intervention techniques?	(a) Usefulness of techniquese; (b) characteristics of educatorsf; (c) appropriate group characteristics (min. and max. age and group size)

Quantitative Survey (Round 2)

Two of the authors (L.G., K.He) reviewed the responses to Round 1. Similar answers were aggregated and duplicates were removed from the data. Disagreements were resolved through team discussions (L.G., K.He, J.H., S.Kn). Through use of this method, the research team created a quantitative questionnaire. To enhance its exhaustiveness, the authors supplemented the questionnaire with items derived from the literature (see Figure 1). We opted for items referring to widely accepted theories of suicidality (as described by Teismann & Dorrmann, 2014), as well the methods of successful programs evaluated with randomized controlled trials (RCTs; Aseltine & DeMartino, 2004; Wasserman et al., 2012). The questionnaire was administered online via SoSci Survey (Leiner, 2016). All 63 members of the original panel were invited to participate. Participants were asked to comment on the correctness and completeness of the questionnaire and to rate the items with regard to different criteria adapted from the ROAMER Expert Survey (Roadmap for Mental Health Research in Europe; Elfeddali et al., 2014) and the indications of Häder (2014) about the conception of expert Delphi surveys (see Table 1).

Figure 1

Overview of the Delphi process.

Statistical Analysis and Re-Rating (Round 3)

The research team analyzed survey responses (medians, interquartile range [IQR], kurtosis) using SPSS (IBM Corporation, version 25). Consensus was defined as an IQR of 1 or below (see von der Gracht, 2012). An IQR of ≤ 1 indicates that 50% of the experts’ responses are within an interval of one point of the Likert scale, whereby the median indicates which point of the scale contains the majority of answers. Participants received the questionnaire in PDF form via e-mail. Each participant received personalized feedback for each of the items where consensus was not reached, and was asked to re-rate those items on the same criteria and scales used in Round 2.

After completion of Round 3, IQRs and medians were computed again to determine consensus, while the Wilcoxon matched-pairs signed-ranks test was used to determine stability. This is a non-parametric equivalent of the paired t test and is used to test for a difference in the mean tendency (median) of paired observations (here: pairs of responses from Rounds 2 and 3; Trevelyan & Robinson, 2015). To avoid unnecessary repetitions, the iterative process was terminated when items reached either consensus or stability (cf. von der Gracht, 2012). Items for which the participants reached consensus and had a median higher than the middle value of the scale were used for content analysis (L.G., S.Kn) to delineate recommendations for school-based awareness programs.

Results

Panel Description

Of the 63 international experts approached, 36 did not respond to the invitation, and four declined participation due to lack of time or because they did not feel they had enough expertise. A total of 23 participants took part in at least one round of the survey: 19 of 63 (30.2%) participated in the first, 16 of 63 (84.2% of Round 1 participants) in the second, and 13 of 63 (81.3% of Round 2 participants) in the third round of the survey. Participants in the first round originated from 12 countries, while participants in the second and third rounds originated from eight (overall: Australia, Austria, Canada, Germany, Iran, Israel, Italy, Hungary, The Netherlands, Norway, Sweden, the United Kingdom, and the United States). Three experts were contacted because of their specific knowledge as leaders of at least one awareness program, and all of them participated in all three rounds. The other participants were authors of influential articles on suicidality and had an H-index > 10 on the Web of Science (range = 14–99).

Delphi Process

Figure 1 summarizes the Delphi process.

Given the small number of items that did not reach the criteria of stability or consensus in Round 3 (six items, 2.8% of total) we decided to terminate the survey, as a further round would not have added substantial findings.

Contents and Favorable Outcomes

In the quantitative survey, the experts were asked to rate the effectiveness of awareness programs when specific types of content were included (30 items). Only one item was rated as very effective for suicide prevention (Mdn = 5, IQR = 1), namely, “Information about where to find help (in the community, at school).” A further 27 items were considered somewhat effective (Table S1 in ESM 1).

We also asked participants to rate the importance, effectivity, and feasibility of outcomes of awareness programs. Three out of 38 outcomes related directly to suicide (reduction of severe suicidal thoughts, fatal and nonfatal suicide attempts); the others were secondary outcomes. The panel expected awareness programs to effect suicide-related outcomes to a small extent (Mdn = 2, IQR = 1). The five items chosen by at least one third of the participants as the most important outcomes included the prevention of suicide attempts and the increase of both help-seeking and peer support. Ten secondary outcomes were identified to be both effective and feasible for suicide prevention in the long term (Mdn ≥ 3, IQR ≤ 1; see Table S2 in ESM 1).

A content analysis of recommended types of content and outcomes resulted in eight central goals for awareness programs: (1) fostering help-seeking (attitudes, behaviors); (2) improving peer support for suicidal youth; (3) improving mental health literacy including knowledge on mental health, mental health disorders and their treatments; (4) education about suicidality (warning signs, real experience reports, false beliefs) and fostering helpful attitudes toward it; (5) informing about issues related to suicidality (e.g., bullying, risk-taking behavior); (6) improving life skills (coping with stress, communication, problem-solving); (7) reducing stigma toward mental health disorders and help-seeking; and (8) reducing risk factors for suicide attempts (hopelessness, social isolation).

Prevention of Adverse Effects

We initially formulated 16 general statements regarding the safety of awareness programs. Although participants argued in favor of school-based awareness programs, they highlighted the need for both adapting these programs to the audience and implementing plans to prevent adverse effects (see Table S3 in ESM 1). In the subsequent rounds, the panel agreed that, “The benefits of suicide prevention programs outweigh the unanticipated consequences” (Mdn = 3, IQR = 1). Of note, two items suggesting that talking about suicide with young people could increase suicidality did not reach consensus, although their median suggested rejection by most of the participants (Mdn = 2, IQR = 2). A further 27 items about precautions were rated as very or moderately important for preventing adverse effects (Mdn ≥ 3, IQR ≤ 1; see Table S4 in ESM 1). Interestingly, to describe suicidality as a symptom of psychopathology (Mdn = 2.5, IQR = 2) and to focus on biological causes of mental health disorders (Mdn = 2, IQR = 2) were rated as being not so important by many of the participants.

Overall, participants suggested to: (1) embed suicide prevention in more general prevention programs; (2) develop, evaluate, and choose programs according to adequate scientific standards; (3) follow guidelines for safe communication about suicide; (4) facilitate access to treatment for participants; (5) prepare long-term action plans to react to suicidality in the schools; (6) train school personnel and parents to recognize and react to suicidality; (7) create an appropriate setting (trained personnel, no punctual intervention); (8) include types of content considered effective for suicide prevention.

Formats and Techniques

The appropriate age range of the target population was estimated to be between 12 (M = 12.2, SD = 2.2) and 17 years (M = 16.5, SD = 6.9), with a group size between 8 (M = 8.4, SD = 5.8) and 20 individuals (M = 19.2, SD = 8.9). Among 17 intervention techniques, participants rated videos, skills trainings, signalizing the presence of gate-keepers, web-based self-management components, and group discussions as the most useful techniques (Mdn ≥ 4, IQR ≤ 1; see Tables S5 and S6 in ESM 1). The program should preferably be led by school psychologists, psychologists, or trained teachers (mean ranks: 1.9, 2.9, and 4; school social workers and school guidance counselors were not included in response formats; see Table S7 in ESM 1).

Discussion

Using a three-stage expert Delphi survey, we developed recommendations on content types, outcomes, formats, and techniques of awareness programs, as well as precautions to prevent negative effects. These findings aim to inform the development, refinement, and dissemination of school-based awareness programs for youth suicide prevention.

In this study, participants rated the prevention of suicide attempts and increasing help-seeking behavior as pivotal targets for prevention programs. This is in line with the idea that improving access to treatment for mental health disorders could be one of the most powerful suicide prevention strategies (Hegerl & Heinz, 2018). Mental health literacy, favorable attitudes toward help-seeking, and communication skills are antecedents of help-seeking behaviors (Rickwood et al., 2005). Hence, the inclusion of this kind of content could be an important step to lower the barriers for access to mental health care. However, further research is needed to determine how awareness programs can promote actual help-seeking behavior (Platt & Niederkrotenthaler, 2020).

The expert panel also suggested that awareness programs should educate young people to react helpfully when peers shows signs of suicidality. This seems particularly important since teenagers often disclose suicidal thoughts only to their friends (Rickwood et al., 2005). The “Teen Mental Health First Aid” program, a program teaching young people to react appropriately to peers with mental health problems, has already been shown to increase the quality of support intentions toward suicidal peers (Hart et al., 2020). Future studies could investigate whether programs focusing on peer support skills can also directly reduce suicide attempts and suicidal thoughts in participating schools. Furthermore, the expert panel recommended to “… embed suicide prevention in more general mental health fostering programs.” Future studies should consider the effects of more general prevention programs (e.g., programs educating about mental health) on suicide attempts and suicidal thoughts.

Overall, most of the survey participants supported the use of school-based awareness programs, stressing that the positive effects outweigh potential negative effects. Interestingly, some of the experts in our study did not support the suggestion to depict suicidality as a symptom of psychopathology, compared with a more normalizing explanation that suicide is the result of extreme stress. This may reflect the assumption that potential stigmas could dissuade young people from talking about their suicidal thoughts. However, Ciffone (2007) did not find any decrease in help-seeking attitudes among those who came to view suicide as a symptom of psychopathology. In fact, Ciffone (2007) asserts that associating suicide with psychopathology could increase the cognitive dissonance in people who were otherwise complacent with their own suicide thoughts, thereby increasing help-seeking. Future studies could investigate which of these two causal explanations is less adverse and more effective.

Nevertheless, the most common recommendation among panel members to minimize any potential adverse effects was to implement evidence-based prevention methodologies that are innocuously effective. The panel also suggested adopting multiple prevention strategies at schools, for example, developing school protocols and educating parents and teachers. Research shows multilevel interventions generally outperforming single interventions (Hofstra et al., 2019). Future studies could test whether this is also true for school-specific interventions.

According to the expert panel, the ideal age for awareness programs should be between 12 and 17 years. Since most prevention programs that have been implemented to date have been targeted at 14- to 16 year-olds, it may be necessary to additionally develop and test programs for younger people.

Limitations

Recommendations from this expert Delphi survey should be interpreted against some caveats. First, these results depend on panel formation and on information upon which participants based their judgment. The panel represents only a small number of experts in the field of suicide prevention. The research team also chose not to provide literature to the panel prior to the survey, mostly because the literature found failed to address the majority of our research questions. Thus, participants’ answers are based on their own knowledge and experience. Second, a systematic literature search or a focus group prior to the survey could have improved its exhaustiveness. Third, for economic reasons, the research team decided to let experts re-rate only those items for which participants did not reach consensus. It is possible that a new rating of all items would have changed the results. Fourth, the inclusion of practitioners and stakeholders with experience in suicide prevention could likely add to our findings.

Conclusion

This Delphi survey is, to our knowledge, the first of its kind to explore content, goals, and methods most suited for effective and safe awareness programs for youth suicide prevention. The improvement of help-seeking and peer support were recommended as key elements of awareness programs. A methodologically sound and long-term evaluation of programs, in combination with long-term interventions in schools that help facilitate access to mental health care for students in need, could further help to prevent the occurrence of negative effects.

The electronic supplementary material is available with the online version of the article at https://doi.org/10.1027/0227-5910/a000783