Engaging primary care professionals in suicide prevention: A qualitative study.

In health systems with strongly developed primary care, such as in the Netherlands, effectively engaging primary care professionals (PCPs) in suicide prevention is a key strategy. As part of the national Suicide Prevention Action Network (SUPRANET), a program was offered to PCPs in six regions in the Netherlands in 2017-2018 to more effectively engage them in suicide prevention. This implementation study aimed to evaluate to what extent SUPRANET was helpful in supporting PCPs to apply suicide prevention practices. From March to May 2018, 21 semi-structured interviews have been carried out with PCPs and other non-clinical professionals from SUPRANET regions in the Netherlands. Verbatim transcripts were analysed using the grounded theory approach. Data was structured using the Consolidated Framework for Implementation Research, which enabled identifying facilitating and challenging factors for PCPs to carry out suicide prevention practices. An important challenge included difficulties in assessing suicide risk (intervention characteristics) due to PCPs' self-perceived incompetence, burdensomeness of suicide and limited time and heavy workload of PCPs. Another important limitation was collaboration with mental health care (outer setting), whereas mental health nurses (inner setting) and SUPRANET (implementation process) were facilitating factors for applying suicide prevention practices. With regard to SUPRANET, especially the training was positively evaluated by PCPs. PCPs expressed a strong need for improving collaboration with specialized mental health care, which was not provided by SUPRANET. Educating PCPs on suicide prevention seems beneficial, but is not sufficient to improve care for suicidal patients. Effective suicide prevention also requires improved liaison between mental health services and primary care, and should therefore be the focus of future suicide prevention strategies aimed at primary care.

Introduction

Suicide is a major health issue. The WHO estimates that 800,000 people worldwide die as a result of suicide every year, which comes down to 2,192 suicides every day [1]. Suicide is a complex phenomenon in which social, cultural and biological factors interact [2]. Therefore, multilevel suicide prevention approaches are preferred above single, standalone measures [3-6].

The European Alliance Against Depression (EAAD) is an example of a multilevel approach. It was founded in 2004 with the purpose of creating a network of countries that have implemented action-focused, community-based interventions to treat depression and prevent suicides [7]. This approach was first tested in 2000 in a region in Germany (Nuremberg), where the total number of suicidal acts decreased by 24% compared to a control region [8]. The model has since been implemented in over 115 regions worldwide [9]. The rationale of the EAAD is that the various levels, including primary care, general public, community facilitators and high-risk groups, interact to create a synergistic and catalytic effect [10]. In the Netherlands, the model focuses on suicide prevention alone and is therefore named Suicide Prevention Action NETwork (SUPRANET). SUPRANET was initiated by 113 Suicide Prevention, the national suicide prevention centre, as part of the national agenda for suicide prevention commissioned by the Ministry of Health [11].

Although all levels of EAAD are relevant for suicide prevention, specific attention is given to primary care. Primary Care Professionals (PCPs) include both General Practitioners (GPs) and their Mental Health Support Staff (MHSS); the latter is a relatively new profession and refers to professionals who offer therapy sessions to primary care patients with mental health, psychosocial or psychosomatic complaints [11]. GPs are often in contact with patients shortly before they engage in suicidal behaviour [12-14]. Additionally, in many health care systems, among which the Netherlands’, GPs function as gatekeepers to identify and refer suicidal patients [15]. Within SUPRANET, a program was developed to support PCPs carrying out evidence-based suicide prevention practices. It contains among others a training to increase their ability to explore and detect suicidal feelings and they are encouraged to improve continuity of care by enhancing collaboration with specialised Mental Health Care (MHC) and other health or community care organisations.

Supporting primary care is among the most effective suicide prevention strategies [5, 16, 17]. However, implementation of interventions in primary care is challenging [9, 18, 19]. An often-used framework to address implementation challenges is the Consolidated Framework for Implementation Research (CFIR). This comprehensive framework describes factors that are important in implementing and evaluating complex interventions. It consists of five domains (characteristics of the intervention, outer setting, inner setting, characteristics of individuals, and process of implementation) which interact and determine effectiveness of implementation together [20]. Using a qualitative design, we conducted this implementation study to evaluate to what extent SUPRANET was helpful in supporting PCPs to apply suicide prevention practices. These insights will be used to engage PCPs more effectively in suicide prevention by improving the use of SUPRANET.

Methods

Design

Semi-structured interviews based on a topic list were used to gather the perceptions and experiences of PCPs with regard to SUPRANET. The CFIR model was used to structure and organize the data. In the present study, intervention characteristics refers to PCPs’ experiences with applying suicide prevention practices such as exploring suicidal feelings. Outer setting refers to the level of collaboration with MHC and in inner setting we discuss the role of MHSS with regard to suicide prevention. The process of implementation describes how the implementation strategy, SUPRANET, was received by PCPs.

SUPRANET for PCPs

In 2017, SUPRANET was implemented in six pilot regions in the Netherlands. The suicide prevention training was offered to both GPs and MHSS between June 2017 and May 2018. In total, 67 PCPs (49 GPs and 18 MHSS) from 23 GP practices completed the suicide prevention training before May 2018 (the study deadline). PCPs were additionally prompted to work on continuity of care by strengthening their collaboration with social community teams, emergency rooms, and MHC services [11]. Additional materials, including the module on suicide and medication, the suicide prevention guide, flyers and posters and a checklist, were provided to the PCPs (see Table 1).

Table 1

Various elements of SUPRANET for PCPs.

Component	Content
Suicide prevention training	The training consists of a theoretical section covering the epidemiology, suicide behaviour, and process of suicide, the Chronological Assessment of Suicidal Events (CASE) methodology, and treatment and referral. The tuition is interspersed with interactive exercises focused on connecting with feelings of despair, involving relatives, and diagnosing. The training is provided by experienced trainers from the Dutch College of General Practitioners and is accredited for four hours.
Continuity of care	Improving collaboration with social community teams, emergency rooms, crisis and MHC services by organizing meetings to discuss issues and make agreements about the treatment policy for suicidal patients.
Module on suicide and medication	A two-hour (accredited) individual e-learning or a group-based pharmaceutical therapeutic audit session about the role of medication in suicide prevention.
Suicide prevention guide	Contains a summary of the most important information from the suicide prevention training and the multidisciplinary guideline for the diagnosis and treatment of suicidal behaviour, and includes a triage tool to assist in referring suicidal patients.
Flyers and posters	Flyers and posters are aimed at patients and relatives, to encourage them to talk about and seek help for suicidal feelings.
Checklist	The checklist can be completed after consultations with (possible) suicidal patients. It includes items such as assessment of suicidal feelings, rumination, and the concreteness of suicidal plans.

Study participants

We used convenience sampling and the snowball technique to recruit study participants from the PCPs within the SUPRANET regions who completed the suicide prevention training. We reached out to PCPs directly or to contact persons in GP practices via email and by phone. In regions where implementation was lagging, i.e., where no PCPs were being trained before the study deadline, professionals who were involved with the implementation of SUPRANET were approached for an interview. In total, 18 PCPs and three non-clinical professionals (two project leaders and one trainer) participated in this study. Financial compensation was available for PCPs who took part in SUPRANET.

Data collection

A topic list was used to guide the semi-structured interviews (See S1 Material). This instrument consists of a list of topics and prompts based on literature [21-23] and discussed within the research group (AK, DB, RG, EE). The rationale for this approach is that it is best suited to capture the perceptions and experiences of participants [24] and it leaves space for participants talk freely and bring up topics or issues themselves. The topic list was compiled to explore PCPs’ perceptions and experiences with SUPRANET, suicide assessment and management in general and their views about liaison with other services. After the first couple of interviews were conducted, we reviewed the topic list and made some minor changes. Instead of focusing on the specific elements of SUPRANET, we focused on SUPRANET as a whole.

Interviews started after the participants had given consent for audio-taping and transcribing the interviews verbatim. Anonymity was guaranteed throughout the entire process. The interviews were conducted by one researcher (EE), who had completed certified training in qualitative research methods including interview skills, in the period October 2018 to March 2019. Sixteen interviews were held in a face-to-face setting. The remaining five interviews were, for logistical reasons, conducted over the phone. The interviews lasted from 21 to 64 minutes.

Data analysis

The transcripts were analysed using the Grounded theory approach. This approach offers a systematic and rigorous process of data collection and analysis, thus facilitating in-depth study of phenomena with the aim of constructing theory from the data [25, 26]. Because of the inductive code structure that evolves from this process, it is most suitable for reflecting the experiences of participants [27]. The Grounded theory approach includes a step-by-step coding procedure, consisting of open coding, axial coding, selective coding reduction, and integration. During the open coding phase all transcripts were read thoroughly by two researchers (EE and a research assistant) and codes were allocated intuitively. Disagreements were discussed until consensus was reached. Then, codes were clustered into themes and categories in a thematic map. Eventually, codes and themes were fitted again onto the transcripts. Finally, during selective coding, underlying differences and similarities were studied, explanations were sought, and literature was integrated. The process of analysis was supported by an experienced qualitative researcher (AK) to ensure validity and reliability. MAXQDA software was used for data analysis [28]. The consolidated criteria for reporting qualitative research (COREQ) was used as checklist for explicit and comprehensive reporting of qualitative data [29].

Ethical approval

According to the Dutch Medical Research Involving Human Subjects Act, this kind of observational study is exempt from ethical review, since the participants were not subjected to any procedure nor were they asked to follow rules of behaviour. Nevertheless, we followed standard ethical procedures by obtaining informed consent from the participants and guaranteeing confidentiality and anonymity throughout the research process.

Results

In this qualitative evaluation study, we interviewed in total 21 participants of whom 13 (62%) were GPs, five (24%) were MHSS and three (14%) were non-clinical professionals involved with the implementation of SUPRANET (“Other”) (see Table 2).

Table 2

Characteristics of the participants.

	Male	Female	Total
GP	8	5	13
MHSS	2	3	5
Other	2	1	3
Total	12	9	21

GP = General Practitioner, MHSS = Mental Health Support Staff

Intervention characteristics

Barriers with regard to assessing suicide risk in the general practice

Many PCPs brought up issues concerning the complexity and unpredictability of suicidality when exploring the issue with patients. They argued that suicidal ideation is difficult to recognize, that it exists in various forms, and that a lot of factors are involved in the suicidal process. PCPs also reported about patients who they felt use suicidality as a strategy to attract attention or as a means of manipulation, making it more challenging to differentiate between actual suicidal patients and patients who may need other forms of care. Additionally, some patients reportedly felt inhibited about discussing mental issues and, therefore, whether consciously or not, presented to their GP with physical complaints. PCPs found it hard to discuss suicidal feelings with these patients straight away and argued that it could take multiple consultations before crossing that bridge.

PCPs said that most patients did not give out distinctive signals indicating suicidality, nor did they proactively disclose their suicidal feelings. Therefore, they argued, it is necessary to actively explore suicide risk. However, many found it hard to determine when to explore this. Most PCPs reported they did not follow a systematic approach for assessing suicidal feelings, but relied on their gut feeling or on the course of the consultation when deciding to explore these thoughts. All PCPs agreed with the recommendation of the multidisciplinary guideline for diagnosis and treatment of suicidal behaviour, which states that suicidal feelings should be assessed during every depression-related consultation, although some argued that it is difficult to apply in everyday practice.

PCPs reported that patients’ suicidal behaviour strongly impacts their professional life. PCPs who experienced one or multiple fatal suicides in their career stated that they carried this experience with them for the rest of their life, especially if they felt that they did not handle the situation properly, which may lead to feelings of self-doubt and failure.

“I have lost someone who was in care in an MHC institution but who had also visited me just before. She jumped from a building […]. I felt like maybe I should have… even if she had come to me about her little toe, maybe I should have kept asking further. Asking her how she was really doing. I failed in that.” (R20)

Although PCPs often sympathized with their patients when they experienced suicidal thoughts, they could also be perceived as a burden, especially when patients were in suicidal crisis and could not be left alone.

“I once had an experience where a person thought this was the hospital. This man was so suicidal that I just stayed with him until the psychiatrist arrived in the evening. In that case, you get sort of stuck with the patient. I thought, if I let him walk out, he’ll jump straight out in front of a car.” (R4)

To address the suicidality of patients can impose a heavy burden on PCPs, who can feel pressured, stressed, and sometimes even personally responsible for their patient’s life.

Of the PCPs, especially GPs reported an enormous workload and very limited time. Not only for consultations with patients but also for extra activities in their GP practice, such as taking part in training- or research. It also played a role in the assessment of suicidal thoughts during consultations:

“When someone in the waiting room has chest pain and a suicidal or depressed patient who has an appointment with an MHC specialist tomorrow consults me, I decide not to ask any further. This might not be good for this patient, but overall it might be for the best.” (R20)

However, most PCPs did not agree with this statement and argued that patients who consult during surgery hours are generally stable and that medical assistants will alert them when this changes. Besides, there are more options than either discussing the matter right away or not discussing it at all. Most PCPs argued they prioritized their patient’s mental wellbeing over their schedule.

“A couple of times, it has happened that consultations take a bit longer, which is really annoying if it’s scheduled for 10 minutes. But in the end, these patients say, ‘Thank you, thank you for taking the time.’ And then, those 10 minutes extra are suddenly a great gift.” (R18)

Outer setting

Collaboration with MHC institutions

Although it varied per region, many PCPs reported issues with regard to collaborating with MHC services. They argued that MHC services operate in isolation and fail to communicate information about patients. Another frequently mentioned issue was difficulties with access to MHC:

“Sometimes you have to pull some serious strings to get someone reviewed or referred. That is difficult. Access is difficult.” (R16)

This resistance was especially noticeable when patients were not in severe crisis, but when the PCPs were worried anyway and required support.

Further, many PCPs reported that processes are very time consuming. Speaking to a professional on the phone takes a long time, admitting a patient into care, starting treatment, or the provision of feedback are inordinately time consuming. While PCPs feel, especially when they refer a patient to the crisis service, that the situation is urgent and requires a fast response.

Additionally, PCPs expressed a need for transparency regarding the methods and procedures in MHC. Some stated that they did not know what happens with a patient during treatment in MHC.

“If you send patients to MHC, I sometimes say, ‘They disappear into a black box. That box is shaken a couple of times and at some point, they fall out.’ I never know what happens to my patients in the meantime, except maybe that they have been relocated six times. […] Sometimes I question whether–with all due respect–crisis services provide better care than I do.” (R2)

Another concern, which is also hinted at by the PCP from the quote above, is the quality of care. Patients are being transferred frequently and see many different health care professionals, who do not always follow the treatment plan that was agreed upon or sometimes not even showing up for an appointment. This kind of behaviour may evoke feelings of rejection and feeling rejected by the very organization that is supposed to provide help, hinders recovery even more.

“Crisis services arouse the feeling that patients experienced before: rejection. They already feel rejected and then they get it all over again.” (R9)

PCPs found the service provision of MHC institutions was substandard. They expressed the desire to collaborate more closely, including having the opportunity to call and discuss patients or to send them in for review, while patients remained treated in primary care. They want to work together, not to shuffle patients back and forth.

“It would be very helpful if they would welcome us in a friendly way, not just the patients, but us too as fellow caregivers. Because we can get at ease from these conversations, ‘Have you already thought of this or that?’. That can make a big difference. Often, we don’t need crisis services to take over but just to provide backup.” (R9)

Most PCPs were very understanding about the situation of the MHC services, which had suffered budget cuts, staff shortages, and onerous policy changes. Nevertheless, they were unsatisfied with the current collaboration and insisted on improvement. Most PCPs reflected on their own practice as well, arguing that they could also improve, but that this too required collaboration and communication.

Inner setting

Role of the MHSS

GPs reported that the MHSS was an enormous relief in terms of providing care for suicidal patients. MHSS are trained to support patients with mental health problems and they have more time than GPs to spend with patients, whereas GPs cannot always provide the close monitoring that these patients may require. Some GPs even argued that MHSS should be given more responsibilities with regard to suicide prevention.

“I think you should make people aware that MHSS play an important role in this; they may even be more important than GPs. They often have more time, expertise, and experience than GPs in dealing with these issues. [This helps] especially when you are kind of insecure as a GP.” (R17)

However, all participants argued that GPs have a distinct part to play in suicide prevention, because, if GPs do not recognise patients’ psychological complaints, patients may not get a referral at all to neither MHSS nor MHC services.

Implementation process

SUPRANET for PCPs consists of a face-to-face training and some additional components. Of all the PCPs who participated in this study, very few had experienced all the components; only two were involved in improving continuity of care and only some used the additional materials. All PCPs participated in the suicide prevention training.

Engaging GP practices in SUPRANET

Regions were responsible for contacting and inviting GP practices to participate, which was a challenging task because of PCPs’ limited available time. One PCP questioned whether the project attracted the right kind of participants using this strategy.

“Only PCPs who are interested in the topic, will sign up for such a program. They often have a background in or affinity with the topic, and, therefore, perform better when faced with these problems during a consultation. The ones who need it the most, who don’t have good communication skills or an affinity with the problem, will not sign up voluntarily”. (R19)

Therefore, this PCP suggested offering the training during the contracted training hours of GP practice staff, so that it is harder for PCPs to avoid the training and it might facilitate implementation.

One of the non-clinical SUPRANET professionals argued they could have used more support during the recruitment phase. When this region involved MHC services to engage GP practices things started to move. Various reasons were offered for this effect: PCPs may favour being approached by other medical doctors, MHC services offered the training for free, and the training was reduced to one or two hours instead of four. An advantage of this approach was instant contact between primary care and MHC, which also facilitated communication and collaboration in daily practice. However, this approach also had some important disadvantages, since there was less control and continuity with regard to the content of the training.

Leadership of SUPRANET

In most regions, a PCP, often a GP, functioned as implementer and spokesperson for SUPRANET. This PCP was responsible for bringing together colleagues from various practices and logistically organising the suicide prevention training. In addition, some of these PCPs were also involved in the continuity of care aspect of SUPRANET. This latter demanded a lot of time and energy and these PCPs argued they expected more support and cooperation from both MHC services and 113 Suicide Prevention. In the absence of this, one PCP felt isolated and became demotivated.

“I’ve sort of lost my motivation. […] I don’t see it happening anymore. I have really put a lot of time into it. […] It feels like it was my request to start this project, but that is not the case. We have been asked by 113 Suicide Prevention, […] and we said yes, but on the assumption that MHC services were also taking part”. (R2)

SUPRANET as strategy for applying suicide prevention practices

Almost all PCPs evaluated the suicide prevention training positively. Although they stated it did not offer any ground-breaking insights, the majority thought it had increased their knowledge about and awareness of suicide prevention. In addition, some argued that SUPRANET had made them feel better equipped to assess patients’ suicide risk:

“Since the training, I ask more often and more targeted questions about people’s emotions and their suicide risk, sometimes very directly, like, ‘Would you take your life?’ […] I did that before as well, but because of the training you become aware of certain aspects of your practice you can improve.” (R1)

Further, the PCPs thought it was valuable to discuss clinical cases with colleagues, since there is usually no time for this during practice hours. They also appreciated being able to practice communication skills during the training and learned, for example, the importance of making actual contact with patients:

“Asking about the patient’s feelings of despair and just carefully listen, without thinking of a solution straight away, were two eye-openers, which I applied the next day. […] and it worked. The lady said: ‘I am so happy that I could tell my story, that you just listened to me’. So, I thought, ‘This works well’.” (R18)

However, there was also some critique on the training, with regard to both content and form. Some PCPs wished the training included more new scientific knowledge and insights. One PCP reported that he would have liked to learn more about risk groups and when to discuss suicidal feelings with patients, instead of how. Other suggestions included incorporating more information about discussing suicidal feelings with specific subgroups, such as children and adolescents, patients with different cultural backgrounds or bereaved family members.

Initially, PCPs thought that conducting four hours of training outside office hours would be too demanding. Afterwards, most were content with it and considered it as something you should prioritize.

“I get it, people want it to be as quick and efficient as possible, but, in terms of effectiveness I think this [the duration] was good. I understand that people want it shorter; everybody is busy, they always are. But I think you should make time for it. There are certain things you just have to make time for.” (R19)

However, since PCPs initially felt resistance towards the duration, they suggested to shorten the training or to split it up into two parts: an individual online training complemented with a face-to-face group training to practice communication skills. The PCPs also strongly recommended follow-up training, since they believed it is not only important to raise awareness but also to maintain it.

The additional materials (checklist, information material, flyers and posters and information module on suicide in relation to medication) were rated by some participants as helpful. Some argued, for instance, the checklist served as a reminder for asking all the relevant questions. Others thought the information material was useful for disseminating knowledge to colleagues. The poster and flyers were, to the PCPs’ knowledge, never used by patients as an encouragement to discuss suicidal feelings. Although not many PCPs completed the module on medication and suicide because of the strict scheduling of training time within their GP practice, they thought this module would be very relevant because not all PCPs have the same knowledge with regard to medication. The continuity of care was assessed as an important component of SUPRANET, which required a lot of improvement, though hardly any PCPs experienced this due to lagging implementation.

Discussion

In this qualitative study, we evaluated part of a multilevel suicide prevention program (SUPRANET), which was implemented in various regions in the Netherlands, as strategy for applying suicide prevention practices in primary care. The CFIR was used to structure the data and to gain insights in facilitating and limiting factors of SUPRANET. All but one domain (individual characteristics) from the CFIR were explicitly addressed. Since this study was based on PCPs’ perceptions and experiences, individual characteristics are intertwined with other domains, particular with intervention characteristics. This study found that there were important barriers to the assessment of suicide risk in patients (intervention characteristics) because of PCPs’ self-perceived incompetence, burdensomeness of suicide and their lack of time and workload. The relationship with MHC was an important limiting factor from the outer setting and with regard to inner setting, MHSS had a positive influence on implementation. Although there were some difficulties in the process of implementation, SUPRANET itself was perceived as useful strategy to improve suicide prevention practices.

Persistent challenges for suicide prevention include the lack of competence to assess suicide risk. In the literature, PCPs also reported challenges with regard to exploring suicidal feelings, because of the perceived reluctance of their patients to disclose suicidal ideation [30] and because many believed that suicide is difficult to predict and prevent [31, 32]. Lack of time is an universally acknowledged barrier to suicide prevention [31]. The present study found that time and workload were limitations for both participating in SUPRANET and for assessing and managing suicide risk with patients.

Hegerl et al. [9] also described in their overall evaluation of the EAAD model, which is the foundation of SUPRANET, that it required significant effort to train GPs. Eventually, 304 GPs from four countries participated in the training, but only after having made considerable adjustments, such as shortening or obliging the training. Another study reported significant improvement in GPs’ attitudes and confidence straight after this training, but only improved confidence was maintained after three to six months [33]. In Hungary, GPs’ need for referral options was not addressed, which, according to the authors, may explain the lack of sustainable changes [9]. Within SUPRANET, the training had been delivered to 67 PCPs by the time of the study’s deadline (March 2018), but delivery continued within these and other regions. GPs were often reluctant to participate because of a lack of time and a heavy workload. However, as PCPs from this study mentioned, participating is not just a matter of time but also a matter of priority.

In the literature, providing support to PCPs is described as one of the most important suicide prevention strategies [5, 16, 17]. On the other hand, Milner et al. [34] argued that many of the included studies were biased and their results varied by study design and outcome. They recommended more studies with randomized, controlled, and blinded designs, as well as establishing effects on various outcome variables, before widely rolling out GP suicide prevention initiatives. Although this study did not meet these requirements, it showed that PCPs valued the training and that it positively influenced their attitudes, suicide assessment and clinical management skills, effects that have also been reported by previous studies [33, 35–37]. Nevertheless, the present study also shows that exploration of suicidal feelings remains difficult and that providing education alone is not sufficient for effective suicide prevention. This is confirmed by Gask who, with regard to educating GPs to recognize and manage depression, states that providing education alone, although necessary, is not sufficient. It is important to make sure that both the content and the provision method of education material suits people at different stages of readiness [38].

The PCPs in the present study expressed a pressing need to improve access to and collaboration with MHC services. The lack of such collaboration is a common barrier to suicide prevention in countries with a primary health care system. In the United Kingdom, for example, many studies have described the issues that GPs have with MHC services. Among these are the tendency of MHC professionals to minimize GPs’ assessments of patients’ suicidal state. GPs report feeling stuck with patients, because they rarely meet the criteria for review and, therefore, remain in primary care [31, 39]. A recent British study described GPs’ feelings of professional isolation as being “lost in a referral maze” [40 p. 5]. British GPs have also expressed the need to have mental health staff based in GP practices [40], such as the MHSS in the Netherlands’ health care system. In the present study, this was indeed perceived as an immense improvement to the management of suicidal patients and it was suggested to expand their role with regard to suicide prevention.

Limitations

The study had several limitations. Since participation in SUPRANET was voluntary, it is likely SUPRANET attracted especially PCPs with a special interest in suicide prevention. Most of the PCPs of this study indeed reported a special interest or experience in suicide prevention or mental health issues. This inherent bias in the sampling method means that the outcomes of the study may not be generalizable to the entire PCP population. Other factors, difficulties, or needs might be identified among PCPs who do not have an interest or experience in the topic. However, it is a well-known phenomenon in implementation studies that these are often based on early adopters. This does not necessarily mean that the issues that are reported here are not applicable to other PCPs. Another limitation is the number of PCPs participating in this study. We could only invite PCPs who participated in SUPRANET and since not all wanted to be interviewed, we eventually included 18 PCPs. In qualitative designs, it is quite normal to have a small sample size, since the aim is to provide rich insights rather than statistical information. There is no rule for determining the sample size of qualitative studies, but similar issues and themes were identified throughout the interviews, indicating that it was sufficient [41].

Implications for practice

The results from the interviews have been discussed with the implementation team of the 113 Suicide prevention, which will continue to implement SUPRANET in new regions in the Netherlands. Regions will now receive more guidance during the recruitment phase, and they will be encouraged to increase the role of MHSS since that may improve the implementation process and the collaboration with MHC. Besides, regions will be urged to involve MHC services from the beginning. Once started, specific focus will be devoted to the continuity of care element, for which a new format has been created: ‘work session: collaborating and making agreements with MHC’. This was developed by the Dutch College of General Practitioners and consists of meetings with relevant stakeholders within the regional chain of care (GPs, MHSS, MHC services and possibly district teams). These meetings will be chaired by an independent party, who will steer towards mutual agreements about the treatment and management of suicidal patients [42]. Some of the proposed adjustments to the training have been carried out already, such as the development of a two-part training. An e-learning program was developed and launched in October 2019 and is freely available for all PCPs. The e-learning can be complemented with group-based skills training to explore suicidal feelings and discuss clinical cases with colleagues, which is strongly recommended. Given the potential of PCPs and their experienced barriers for effective suicide prevention, we encourage that national governments and professional PCP associations devote attention to suicide prevention in primary care, help addressing these challenges and put efforts into engaging PCPs more effectively.

Conclusion

This study is among the few that describes experiences and perceptions with regard to a suicide prevention program for primary care as part of a multilevel community-based model for suicide prevention. The study provides deeper understanding of how PCPs can be better engaged in suicide prevention. SUPRANET was perceived as useful strategy to improve suicide prevention practices. The training increased PCPs’ awareness and knowledge of suicide prevention, which facilitates the recognition of suicidal patients. Effective suicide prevention, however, also requires improved collaboration between primary care and MHC. More effectively addressing this will facilitate implementation and effectiveness of SUPRANET and further engage PCPs in suicide prevention.

Topic list.

(DOCX)

Click here for additional data file.

3 Jun 2020

PONE-D-20-01479

Engaging Primary Care Professionals in suicide prevention: a qualitative study

PLOS ONE

Dear Dr. Elzinga,

Thank you for submitting your manuscript to PLOS ONE. I have recently taken over this manuscript as Academic Editor and have had the opportunity to review the comments provided by the expert reviewer that the prior Academic Editor invited. I have also had a chance to review your paper as well. As you will see below, the reviewer had a generally positive opinion of the paper but had concerns that are likely addressable. Accordingly, I am inviting you to revise and resubmit your manuscript. Please do know that this is no guarantee of ultimate acceptance and I will send the revised version of this manuscript back to the original reviewer for comments again.

Although all of the reviewer's comments are important to address, there are a few comments in particular that I'd like you to pay attention to. Specifically, it would be good to add more detail on (a) the background of how PCPs are trained in suicide screening/prevention/intervention and (b) what the specific suicide screening/prevention/intervention programs that are implemented are. Additionally, as the reviewer notes, there are existing research frameworks where your study fits well (e.g,. The Consolidated Framework for Implementation Research) and it would be good to discuss this in your introduction.

Please submit your revised manuscript by Jul 18 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Evan M Kleiman

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. For qualitative studies, PLOS ONE suggests consulting the COREQ guidelines: http://intqhc.oxfordjournals.org/content/19/6/349 to ensure that all relevant information is provided (in this case we would appreciate more information about: the number  and training of interviewers; how participants were selected; if bias issues were considered.

3. Please include the registration number for the clinical trial referenced in the manuscript.

4. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

5. Please include a separate caption for each figure in your manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for the opportunity to review this manuscript. This paper presents the results of qualitative interviews with 21 individuals involved in the implementation of a primary care based suicide prevention program, a timely and important topic. I have several concerns about the manuscript in its current form, detailed below, that I think would be helpful for the authors to address.

While the manuscript is generally well written, additional proofreading by a native English speaker would be helpful.

Perhaps it is just a problem with the proof, but the text in Box 1 appears cut off.

It might be helpful to describe the suicide screening/prevention/intervention practices being implemented as the “interventions” and the additional training and support being offered to the clinicians as the “implementation strategies.” It would also be helpful to understand the intensity of training and supports provided to PCPs. Powell et al’s (2015) refined compilation of implementation strategies paper and Proctor et al.’s (2013) paper on specifying and reporting implementation strategies, both published in Implementation Science, may be helpful in these regards. I think this approach would improve clarity throughout.

The authors are encouraged to consider making the “topic list” used to guide interviews available as supplemental to the paper. What was the rationale for this approach versus a more traditional semi-structured qualitative interview guide? A semi-structured guide has several advantages including ensuring uniform inclusion and sequencing of questions.

If there is any additional demographic information about participants please consider including it, for example participant race, ethnicity, mean age, mean number of years in practice, and information about the types of practices (community clinics vs. academically affiliated centers for example).

Some of the content of the results seems to report perceptions or suggestions made by one PCP. While it is certainly appropriate to provide illustrative quotations from participants, the reader can be much more confident in the trustworthiness of overall themes that emerged based on the perspectives of many of the interviewees and not individual interviewee perspectives. It may be that this is simply an issue with how the results were presented in the paper and not the actual coding process.

What is 113 referring to in the Results? Is this masking a person or program or an actual name of a program?

Consider grounding this work in an established implementation framework, like the Consolidated Framework for Implementation Research (CFIR). Most of the themes fit within established constructs from frameworks like CFIR and organizing and presenting them in this way could be useful. An integrated coding approach (see Bradley) that combines inductive and deductive coding could offer advantages to solely utilizing grounded theory here. It could be helpful to review the CFIR implementation process constructs in particular to see if those may be useful for organizing the presentation of the results around providers experience with implementation. Another suggestion would be to specifically pull out barriers, facilitators, and suggested strategies/changes and present as their own themes. Presently these are presented within other themes.

Finally, I see that e-learning is mentioned in the discussion as a strategy that was suggested by providers, but I don’t recall seeing that presented in the results. I suggest ensuring that all themes commented on in the discussion are reported in the results.

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

17 Jul 2020

Dear Dr. Elzinga,

Thank you for submitting your manuscript to PLOS ONE. I have recently taken over this manuscript as Academic Editor and have had the opportunity to review the comments provided by the expert reviewer that the prior Academic Editor invited. I have also had a chance to review your paper as well. As you will see below, the reviewer had a generally positive opinion of the paper but had concerns that are likely addressable. Accordingly, I am inviting you to revise and resubmit your manuscript. Please do know that this is no guarantee of ultimate acceptance and I will send the revised version of this manuscript back to the original reviewer for comments again.

Although all of the reviewer's comments are important to address, there are a few comments in particular that I'd like you to pay attention to. Specifically, it would be good to add more detail on (a) the background of how PCPs are trained in suicide screening/prevention/intervention and (b) what the specific suicide screening/prevention/intervention programs that are implemented are. Additionally, as the reviewer notes, there are existing research frameworks where your study fits well (e.g,. The Consolidated Framework for Implementation Research) and it would be good to discuss this in your introduction.

Please submit your revised manuscript by Jul 18 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Evan M Kleiman

Academic Editor

PLOS ONE

Dear dr. Evan Kleiman,

Thank you for providing the opportunity to revise the manuscript and considering it for publication in PLOS ONE. We believe that the revisions have improved the manuscript considerably. We have addressed your comments below using italic script.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Thank you, I have adjusted the manuscript according to the style requirements.

2. For qualitative studies, PLOS ONE suggests consulting the COREQ guidelines: http://intqhc.oxfordjournals.org/content/19/6/349 to ensure that all relevant information is provided (in this case we would appreciate more information about: the number and training of interviewers; how participants were selected; if bias issues were considered.

Thank you for this comment. We have carefully read the COREQ guidelines and added information about the number and training of interviewers, participant selection and if biases were considered.

Line 126-128: “The interviews were conducted by one researcher (EE), who had completed certified training in qualitative research methods including interview skills, in the period October 2018 to March 2019.”

Line 105-110: “We used convenience sampling and the snowball technique to recruit study participants from the PCPs within the SUPRANET regions who completed the suicide prevention training. We reached out to PCPs directly or to contact persons in GP practices via email and by phone. In regions where implementation was lagging, i.e., where no PCPs were being trained before the study deadline, professionals who were involved with the implementation of the SUPRANET intervention were approached for an interview.”

Line 471-476: “Since participation in the intervention and evaluation study was voluntary, it is likely that especially PCPs with a special interest in suicide prevention predominated among those who signed up. Most of the PCPs of this study indeed reported they had a special interest or experience in suicidality or mental health issues, for example, by having worked in a MHC institution. This inherent bias in the sampling method means that the outcomes of the study may not be generalizable to the entire PCP population.”

3. Please include the registration number for the clinical trial referenced in the manuscript.

We have not referred to a clinical trial in this manuscript.

4. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

The paper includes anonymized quotes of the interviewees to support the results. It is not possible to make the full transcripts of the interviews publicly available. The interviewees have not given consent for this, and they may contain sensitive or identifiable information. The transcripts are archived on a secured server of 113 Suicide Prevention. Anonymized excerpts of transcripts are available upon request, which can be made via email to the corresponding author (Elke Elzinga; e.elzinga@113.nl) or to the research department of 113 Suicide Prevention (onderzoek@113.nl).

We will update your Data Availability statement to reflect the information you provide in your cover letter.

5. Please include a separate caption for each figure in your manuscript.

We have included captions for all tables and excluded the figure from the manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for the opportunity to review this manuscript. This paper presents the results of qualitative interviews with 21 individuals involved in the implementation of a primary care based suicide prevention program, a timely and important topic. I have several concerns about the manuscript in its current form, detailed below, that I think would be helpful for the authors to address.

Dear reviewer,

Thank you for you carefully formulated comments. We have addressed them in the manuscript and in this letter, using italic script.

While the manuscript is generally well written, additional proofreading by a native English speaker would be helpful.

Thank you for your suggestion, the manuscript is now proofread by a professional language bureau.

Perhaps it is just a problem with the proof, but the text in Box 1 appears cut off.

Thank you for noticing, the box was indeed cut off. We have deleted the box and incorporated it in the introduction.

Line 65-69: “Since 2007, GPs can also refer patients to the Mental Health Nurse (MHN), a role introduced to cope with the increasing demand for mental health issues in primary care [21]. MHNs include professionals with a nursing or psychology background who can provide brief therapy sessions to patients within primary care [10].”

It might be helpful to describe the suicide screening/prevention/intervention practices being implemented as the “interventions” and the additional training and support being offered to the clinicians as the “implementation strategies.” It would also be helpful to understand the intensity of training and supports provided to PCPs. Powell et al’s (2015) refined compilation of implementation strategies paper and Proctor et al.’s (2013) paper on specifying and reporting implementation strategies, both published in Implementation Science, may be helpful in these regards. I think this approach would improve clarity throughout.

Thank you for this suggestion. We have clarified the terms by referring to “intervention” with regard to the entire package, including the suicide prevention training, continuity of care element, module on suicide and medication, suicide prevention guide, flyers and posters and checklist (see table 1 below), and by referring to “implementation strategies” with regard to offering the intervention to the clinicians all over the manuscript. We have also adjusted the table and added extra information about what these elements entail and described the intensity of the elements.

Line 102:

Table 1. Various elements of the SUPRANET intervention for PCPs

Suicide prevention training

The training consists of a theoretical section covering the epidemiology, behaviour, and process of suicide, the Chronological Assessment of Suicidal Events (CASE) methodology, and treatment and referral. The tuition is interspersed with interactive exercises focused on making contact with feelings of despair, involving relatives, and diagnosis. The training is provided by experienced trainers from the Dutch College of General Practitioners and is accredited for four hours.

Continuity of care

CPs are encouraged to arrange meetings with organizations in the regional chain of care for suicidal patients (e.g., MHC institutions, crisis services, or emergency rooms) to discuss issues and to enter into agreements to improve the continuity of care.

Module on suicide and medication

A two-hour (accredited) individual e-learning session or a pharmaceutical therapeutic audit completed with colleagues takes place on the subject of the role of medication in suicide prevention.

Suicide prevention guide

The suicide prevention guide contains a summary of the most important information from the suicide prevention training and the multidisciplinary guideline for the diagnosis and treatment of suicidal behaviour including a triage tool to help in referrals of suicidal patients.

Flyers and posters

Flyers and posters are aimed at patients and relatives, to encourage them to talk about and seek help for suicidal feelings.

Checklist

The checklist can be completed after consultations with (possible) suicidal patients. The checklist includes items such as assessment of suicidal feelings, rumination, and the concreteness of suicidal plans.

The authors are encouraged to consider making the “topic list” used to guide interviews available as supplemental to the paper. What was the rationale for this approach versus a more traditional semi-structured qualitative interview guide? A semi-structured guide has several advantages including ensuring uniform inclusion and sequencing of questions.

Thank you for this suggestion. We have added the topic list in the supplementary files (S1 File. Topic list) and explained the reasons for this approach in the method section.

Line 114-119: “A topic list was used to guide the semi-structured interviews (See S1. Topic list). This instrument consists of a list of topics and prompts based on literature [24-26] and discussed within the research group (AK, DB, RG, EE). The rationale for this approach is that it is best suited to capture the perceptions and experiences of participants [27]. Although we mostly addressed the topics in the given order, if required, we could discuss them in another order, and it leaves space for participants to talk freely and bring up topics or issues themselves”

If there is any additional demographic information about participants please consider including it, for example participant race, ethnicity, mean age, mean number of years in practice, and information about the types of practices (community clinics vs. academically affiliated centers for example).

Unfortunately, there is not. We only have information regarding function and gender (and region, but that cannot be published due to privacy restrictions).

Some of the content of the results seems to report perceptions or suggestions made by one PCP. While it is certainly appropriate to provide illustrative quotations from participants, the reader can be much more confident in the trustworthiness of overall themes that emerged based on the perspectives of many of the interviewees and not individual interviewee perspectives. It may be that this is simply an issue with how the results were presented in the paper and not the actual coding process.

Thank you for your suggestion. The results are not based upon the data from one PCP, but on all 21 participants. We have included numbers of the interviewees to the quotes, hopefully this clarifies that the comments originated from different participants.

What is 113 referring to in the Results? Is this masking a person or program or an actual name of a program?

113 refers to 113 Suicide prevention: the initiator of SUPRANET. We have now addressed this (more clearly) in the method section.

Line 85-86: “SUPRANET was initiated by 113 Suicide Prevention, the Netherlands national suicide prevention centre, as part of the national agenda for suicide prevention commissioned by the Ministry of Health”.

Consider grounding this work in an established implementation framework, like the Consolidated Framework for Implementation Research (CFIR). Most of the themes fit within established constructs from frameworks like CFIR and organizing and presenting them in this way could be useful.

We have carefully read the paper about the CFIR and it seems to provide a very relevant framework for when performing an implementation study. In this evaluation study, where we focused on the perceptions and experiences of PCPs with the intervention, we did not select a framework to guide the data collection and analysis procedure upfront. Therefore, we chose to discuss the CFIR it in the discussion.

Lines 407-426: “Although we did not select a framework a priori, using a framework, such as Damschroder and colleagues’ Consolidated Framework for Implementation Research (CFIR), may aid in the evaluation of complex interventions such as SUPRANET [34]. The CFIR consists of five domains (intervention characteristics, outer setting, inner setting, characteristics of individuals, and process) that interact in various ways and determine the effectiveness of implementation. When applying the CFIR retrospectively, we were able to establish some factors that may have played a crucial role in the effectiveness of the intervention. The suicide prevention training can be seen as the “core component” with an “adaptable periphery” consisting of the other elements and its flexible delivery. According to the CFIR, this facilitates implementation and thus influences effectiveness. Further, peer pressure from other PCPs, perceived patients’ needs, and time constraints can be identified as factors from the outer setting that influence participation. Factors from the inner setting include a positive climate for implementation within the GP practices, shown for example by PCPs who wanted to participate because of experiences with a patient’s suicide (tension for change), because they wanted to develop their skills (positive learning climate), or because of the perceived urgency of the topic (relative priority). This study also indicated that some individuals, notably key PCPs, played an important role in the implementation of the intervention. The same applies in reverse; implementation in certain regions may have lagged behind others because no such individuals were driving the initiative. Although we could apply this framework retrospectively, more or other factors may have come up when applying it from the beginning. Therefore, future evaluation studies are recommended to use this or another framework to guide the evaluation from the start.”

• An integrated coding approach (see Bradley) that combines inductive and deductive coding could offer advantages to solely utilizing grounded theory here.

In their article Bradley et al 2007 focus on qualitative data analysis that has as a goal the generation of taxonomy, themes, and theory germane to health services research. Because of the explorative nature of our research we conducted a grounded theory approach. Our approach was inductive and data-driven, focused upon identifying and discussing the salient themes repeated across and within transcripts. It involves coding and the generation (and interpretation) of broader patterns in data. Our choice is therefore prompted by carrying out the analysis from this point of view.

• It could be helpful to review the CFIR implementation process constructs in particular to see if those may be useful for organizing the presentation of the results around providers experience with implementation.

Thank you for your suggestion. Even though evaluation the implementation process was not the main goal of the paper, implementation-related themes did emerge from the data which we addressed in the chapter ‘implementation strategies’ (lines 165-207). Further, we discussed the application of the CFIR framework onto our results in the discussion. From the implementation construct, we were especially able to establish the role of certain individuals (key PCPs) during the implementation process.

Line 421-425: “This study also indicated that some individuals, notably key PCPs, played an important role in the implementation of the intervention. The same applies in reverse; implementation in certain regions may have lagged behind others because no such individuals were driving the initiative. Although we could apply this framework retrospectively, more or other factors may have come up when applying it from the beginning.”

• Another suggestion would be to specifically pull out barriers, facilitators, and suggested strategies/changes and present as their own themes. Presently these are presented within other themes.

Thank you for this suggestion. However, the rationale of this study was to provide insights into the perceptions and experiences of PCPs with the intervention and suicide prevention in general practice, which will ultimately be used to enhance the intervention and to engage PCPs more effectively in suicide prevention. The topic list that we used in this study was based on this rationale. Although barriers and facilitators emerged, we did not set up this study to focus on those specifically. It should be noted that the barriers and facilitators that are being described in the present study emerged within the context of the themes and are non-exhaustive. From this perspective, we think the chosen structure into the main themes is most suitable.

Finally, I see that e-learning is mentioned in the discussion as a strategy that was suggested by providers, but I don’t recall seeing that presented in the results. I suggest ensuring that all themes commented on in the discussion are reported in the results.

Thank you, we have described this in the result section:

Line 238-239: “Other suggestions included shortening the training or splitting it up into two parts: online individual training complemented with training in a group to practice communication skills.

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Once again, thank you for your comments. We believe that with these revisions the manuscript has improved a lot.

Submitted filename: Response to reviewers.docx

Click here for additional data file.

9 Sep 2020

PONE-D-20-01479R1

Engaging Primary Care Professionals in suicide prevention: a qualitative study

PLOS ONE

Dear Dr. Elzinga,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

As you will see below, the reviewer still has several lingering concerns. I share these concerns as well and am, at this point, uncertain about whether or not it will be possible to revise the paper to correct these issues. I would like to give you another opportunity to address the issues and respond to the reviewer's very detailed feedback. If you are able to do this, there is a chance that the paper may be fit for PLOS One. But, I should say that I will completely understand if you choose to instead submit this paper to another outlet. If you do choose to submit elsewhere, please let me know.

If you do choose to resubmit, please submit your revised manuscript by Oct 24 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Evan M Kleiman

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for the opportunity to re-review this manuscript. I appreciate the detailed rationale provided in the authors’ response to reviews. There are a few remaining issues that warrant consideration.

Minor point- Table 2. I do not think this level of detail about participants is necessary, rather I would suggest reporting in aggregate the n (%) of participants who are male, GPs etc.

There remains some confusion with respect to the intervention vs. implementation strategies. In the way these terms are typically used, the interventions are the evidence based suicide prevention strategies that the PCPs use with their patients. The implementation strategies are the strategies employed in order to get PCPs to implement EBPs for suicide prevention, such as training, consultation, facilitation etc.

Geoff Curran has a paper on the topic that very clearly describes the distinction: Curran, G.M. Implementation science made too simple: a teaching tool. Implement Sci Commun 1, 27 (2020). https://doi.org/10.1186/s43058-020-00001-z

Specifically, it would be helpful to clearly distinguish participants’ reactions to the SUPRANET training/supports (the implementation strategies) vs. their reactions to implementing EBPs for suicide prevention (the intervention).

I think the way the CFIR is integrated at present is confusing. I understand you do not wish to retrospectively apply CFIR to your data and that is fine, however if that is the case then I would not attempt to do that in the Discussion as you did. Instead, I would suggest removing CFIR from the Discussion and incorporating it into the Introduction. I do believe that this is in fact an implementation study- you are studying an effort to train providers to implement suicide prevention practices, exploring PCPs reactions and utilization of suicide prevention practices, etc. In contrast, an intervention study would typically involve studying the impact of suicide prevention practices on patients. Again, Geoff Curran’s paper may be helpful clarification.

Given this, CFIR seems quite relevant. In the Introduction I would suggest you introduce the CFIR as a comprehensive framework that describes the relevant contextual factors that have been shown to be important to consider when planning for the implementation of a health service intervention (e.g., suicide prevention EBPs). Then, I would briefly describe the 5 main CFIR constructs and delineate which you focused on in the present project. For example, Characteristics of the Intervention (e.g., the characteristics of suicide prevention EBPs that may impact their use in a particular setting or by particular providers), Outer Setting (e.g., external factors such as political or policy factors related to suicide prevention, patient populations served by the PCPs), Inner Setting (e.g., what is it about the way clinics are structured or ways in which workflows or availability of behavioral health consultant impacts practice?), Characteristics of the Individuals implementing suicide EBPs (e.g., PCP attitudes about suicide prevention, self-efficacy to implement suicide EBPs) and Implementation Process (e.g., what was the experience of receiving training through SUPRANET like).

I hope this helps to clarify some of my previous comments from the initial submission.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

23 Oct 2020

Dear editor and reviewer,

Thank you for the opportunity to revise the manuscript once more. We have discussed your comments thoroughly within our research team and addressed these in the manuscript. Below, we will provide an answer to the issues one-by-one:

Reviewer #1: Thank you for the opportunity to re-review this manuscript. I appreciate the detailed rationale provided in the authors’ response to reviews. There are a few remaining issues that warrant consideration.

Minor point- Table 2. I do not think this level of detail about participants is necessary, rather I would suggest reporting in aggregate the n (%) of participants who are male, GPs etc.

Thank you for your comment, we have adjusted the table (line 149).

There remains some confusion with respect to the intervention vs. implementation strategies. In the way these terms are typically used, the interventions are the evidence based suicide prevention strategies that the PCPs use with their patients. The implementation strategies are the strategies employed in order to get PCPs to implement EBPs for suicide prevention, such as training, consultation, facilitation etc.

Geoff Curran has a paper on the topic that very clearly describes the distinction: Curran, G.M. Implementation science made too simple: a teaching tool. Implement Sci Commun 1, 27 (2020). https://doi.org/10.1186/s43058-020-00001-z

Specifically, it would be helpful to clearly distinguish participants’ reactions to the SUPRANET training/supports (the implementation strategies) vs. their reactions to implementing EBPs (evidence-based practices) for suicide prevention (the intervention).

Thanks for clearing up the confusion. We have changed this throughout the manuscript; SUPRANET is now described as implementation strategy to help PCPs applying suicide prevention practices. See for example the altered aim on line 75-78:

“Using a qualitative design, we conducted this implementation study to evaluate to what extent SUPRANET was helpful in supporting PCPs to apply suicide prevention practices. These insights will be used to engage PCPs more effectively in suicide prevention by improving the use of SUPRANET.”

I think the way the CFIR is integrated at present is confusing. I understand you do not wish to retrospectively apply CFIR to your data and that is fine, however if that is the case then I would not attempt to do that in the Discussion as you did. Instead, I would suggest removing CFIR from the Discussion and incorporating it into the Introduction. I do believe that this is in fact an implementation study- you are studying an effort to train providers to implement suicide prevention practices, exploring PCPs reactions and utilization of suicide prevention practices, etc. In contrast, an intervention study would typically involve studying the impact of suicide prevention practices on patients. Again, Geoff Curran’s paper may be helpful clarification.

Given this, CFIR seems quite relevant. In the Introduction I would suggest you introduce the CFIR as a comprehensive framework that describes the relevant contextual factors that have been shown to be important to consider when planning for the implementation of a health service intervention (e.g., suicide prevention EBPs). Then, I would briefly describe the 5 main CFIR constructs and delineate which you focused on in the present project. For example, Characteristics of the Intervention (e.g., the characteristics of suicide prevention EBPs that may impact their use in a particular setting or by particular providers), Outer Setting (e.g., external factors such as political or policy factors related to suicide prevention, patient populations served by the PCPs), Inner Setting (e.g., what is it about the way clinics are structured or ways in which workflows or availability of behavioral health consultant impacts practice?), Characteristics of the Individuals implementing suicide EBPs (e.g., PCP attitudes about suicide prevention, self-efficacy to implement suicide EBPs) and Implementation Process (e.g., what was the experience of receiving training through SUPRANET like).

I hope this helps to clarify some of my previous comments from the initial submission.

Thanks for your explanation. We have now introduced the CFIR in the introduction, explained how we apply the model in the method-section and we have organized the results according to the CFIR structure. In the discussion we present the most important findings according to the CFIR (see below).

Introduction (line 70-75)

“An often-used framework to address implementation challenges is the Consolidated Framework for Implementation Research (CFIR). This comprehensive framework describes factors that are important in implementing and evaluating complex interventions. It consists of five domains (characteristics of the intervention, outer setting, inner setting, characteristics of individuals, and process of implementation) which interact and determine effectiveness of implementation together [20].”

Methods (line 83-87)

“The CFIR model was used to structure and organize the data. In the present study, intervention characteristics refers to PCPs’ experiences with applying suicide prevention practices such as exploring suicidal feelings. Outer setting refers to the level of collaboration with MHC and in inner setting we discuss the role of MHNs with regard to suicide prevention. The process of implementation describes how the implementation strategy, SUPRANET, was received by PCPs.”

Results (line 144-331)

see manuscript

Discussion (line 335-344)

“The CFIR was used to structure the data and to gain insights in facilitating and limiting factors of SUPRANET. All but one domain (individual characteristics) from the CFIR were explicitly addressed. Since this study was based on PCPs’ perceptions and experiences, individual characteristics are intertwined with other domains, particular with intervention characteristics. This study found that there were important barriers to the assessment of suicide risk in patients (intervention characteristics) because of PCPs’ self-perceived incompetence, burdensomeness of suicide and their lack of time and workload. The relationship with MHC was an important limiting factor from the outer setting and with regard to inner setting, MHSS had a positive influence on implementation. Although there were some difficulties in the process of implementation, SUPRANET itself was perceived as useful strategy to improve suicide prevention practices. “

Submitted filename: Rebuttal letter.docx

Click here for additional data file.

5 Nov 2020

Engaging Primary Care Professionals in suicide prevention: a qualitative study

PONE-D-20-01479R2

Dear Dr. Elzinga,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Evan M Kleiman

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

9 Nov 2020

PONE-D-20-01479R2

Engaging primary care professionals in suicide prevention: a qualitative study

Dear Dr. Elzinga:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Evan M Kleiman

Academic Editor

PLOS ONE