Development and Preliminary Evaluation of an Education Program for Primary Care Teams on Discussing Firearms Storage Safety with Veterans.

BACKGROUND: Reducing access to lethal means is one of the few empirically supported approaches for lowering suicide rates, and safe firearms storage practices have been associated with reduced risk of death by suicide. Although there is substantial opportunity for primary care to assist in addressing lethal means with veterans, approaches to intervention and educating staff are not well documented. We sought to 1) describe development of an education program for primary care teams to help them discuss firearms storage safety (FSS) with veterans during primary care visits; and 2) conduct a preliminary evaluation of the pilot education program.
METHODS: We used an iterative process involving veterans and primary care staff stakeholders to develop program content, format, and supplemental materials. A grounded theory approach was used to analyze data from focus groups and individual interviews. Following piloting of the program with 71 staff members in two primary care clinics, we analyzed pre- and post-training participant surveys of program satisfaction and attitudes comfort related to firearms safety discussions.
RESULTS: During the development phase, 68 veterans and 107 staff members participated in four veteran focus groups and four primary care focus groups, respectively, and/or individual interviews. The program that was developed, "'Just in Case': Discussing means safety with veterans at elevated risk for suicide," addresses knowledge and skills learning objectives, and includes video demonstrations and skills practice. Survey data obtained just prior to the pilot training sessions showed low self-reported rates of discussing firearms safety with veterans who may be at elevated risk for suicide. Immediate post-training data showed generally high satisfaction with the program and significant improvements in participant self-reported ratings of the importance of, and comfort with FSS.
CONCLUSIONS: This interactive knowledge and skill-based means safety curriculum shows promise as a means for educating primary care staff to deliver messaging about firearms safety to veterans. Additional research is needed to refine and evaluate impacts of this or similar training programs on clinician and veteran behaviors over time.

Background

Veterans are 1.5 times more likely to die by suicide compared to non-veterans. Veterans are also significantly more likely to use firearms as a means of suicide compared to the general population. Reducing access to lethal means is one of the few empirically supported approaches for lowering suicide rates,[3,4-7] and safe firearms storage practices are associated with reduced risk of death by suicide.[5-7]

Means safety counseling approaches have been developed in an effort to reduce deaths by firearms and other suicide methods.[8,9] Means safety counseling rests on the premise that moments of increased suicide risk are often fleeting, and that 1) preventive steps can be taken to reduce the likelihood of making an attempt, and 2) reducing access to the most lethal methods of suicide (eg, firearms) increases likelihood that individuals who do attempt suicide will survive.[10,11] Means safety counseling is considered a clinical best practice for individuals with identified suicide risk, and increasingly supported by multiple national organizations.[13-15] Promising models exist for means safety counseling, such as those incorporated into Counseling on Access to Lethal Means (CALM).[8,16] Typically, means safety counseling consists of a conversation between a counselor and an at-risk individual, often in clinical settings, whereby the at-risk individual is counseled to reduce their access to firearms (often through removal) as well as other lethal means. Bryan recently found that a training program for VHA-based clinical staff led to higher rates of clinician-reported counseling of veterans about firearms safety. Although the study participants included a small proportion of physicians and nurses, the majority of the sample was comprised of social workers and psychologists.

While means safety counseling can theoretically be delivered in any clinical setting, discussions about firearms occur infrequently in non-mental health settings, including primary care.[18-20] Primary care clinicians are often uncomfortable discussing firearms with patients due to perceived barriers, such as lack of knowledge or personal experience with firearms, or concerns about negative impacts to the therapeutic alliance. Traditional means safety counseling can be time intensive, and competing demands during brief appointments also likely limit the ability and willingness of primary care clinicians to discuss firearms.[8,9]

Yet, primary care is a critically important setting for identifying and engaging veterans at risk for suicide. Half of individuals, including veterans, who die by suicide are seen in primary care settings in the month prior to death.[22,23] Most individuals who receive mental health treatment receive it only in the primary care setting. Thus, many patients who might benefit from discussions about firearms storage safety (FSS) in particular will be missed if interventions are not developed specifically for primary care. FSS discussions specifically include 1) advising veterans to keep firearms locked and unloaded when not in use; and 2) identifying additional ways to reduce access to firearms (eg, removal of the firearm) in situations when suicide risk may increase, such as during a crisis. Over the past decade, many healthcare systems, including the Veterans Health Administration (VHA), have implemented standardized approaches to screening for mental health conditions, including depression and post-traumatic stress disorder (PTSD), as well as suicidal ideation. The screening process provides an important opportunity to implement FSS discussions into primary care, allowing for intervention with individuals who may be at higher risk for suicide, but before the individual develops a crisis.

Our research has demonstrated that both primary care teams and veterans support having FSS discussions during primary care visits, with certain caveats.[25-27] The purpose of this manuscript is to 1) describe stakeholder-informed development of an education program to help primary care teams discuss FSS during primary care visits; and 2) present results of a preliminary evaluation of the pilot program.

Methods

Setting and Samples

The goal of the overall project was to develop an education program to facilitate delivery of FSS by VHA primary care teams. While we envision FSS as being deliverable to any Veteran, we specifically sought to help primary care teams focus on delivering FSS to individuals with positive depression or PTSD screens. The first phase of the project, focused on developing the program itself, was designated as a quality improvement activity by VA Portland Health Care System (VAPORHCS) on 6/15/2018 (no reference number). The second phase, which focused on evaluating the pilot program, was approved by the VAPORHCS Institutional Review Board (IRB) on 11/30/2018 (reference #4347); the IRB approved a waiver of written informed consent; all participants were provided an information sheet and gave verbal consent to participate. During the first phase, we worked with veteran and primary care teams to explore attitudes toward FSS, and to gather input on development of the training program. Three local veteran organizations were identified via word-of-mouth, and organization leadership was approached to set up focus group meetings. All members of these organizations were invited to respective group meetings to create a convenience sample of veterans. A fourth group of veteran consultants, identified via communications with leaders of the above three groups, was created to provide input into this specific project. We also conducted focus groups with primary care teams at three VA community-based outpatient clinics (CBOCs) and one hospital-based primary care clinic. Due to firearm ownership being a sensitive issue, we intentionally did not ask individuals participating in focus groups about their own firearms ownership. However, prior research indicates that approximately half of veterans own firearms, and we presumed or observed that the majority of participants either owned firearms or had positive attitudes toward firearms ownership, as indicated by self-disclosure during sessions and the general absence of comments indicating negative attitudes toward gun ownership. Five large primary care clinics associated with VAPORHCS were also invited to participate in sessions; we met with the first four clinics that responded. Finally, six veterans and five primary care providers completed individual, semi-structured interviews following the focus group meetings. The total number of clinicians who participated in group and/or individual sessions was 107 and the total number of veterans who participated in group and/or individual sessions was 68.

Development of the Training Program

The methods and findings from the focus group meetings and individual interviews with veterans and staff have previously been published.[25-27] Table 1 shows a summary of findings regarding attitudes and advice given by these stakeholders which specifically informed development of the training program.

Table 1.

Summary of themes from development Phase.

ATTITUDES—VETERANS	ATTITUDES—PRIMARY CARE STAFF
Discussing firearms safety is acceptable and necessary, even if discussions are uncomfortable. Veterans support a team-based approach because staff other than providers may have more rapport and contact with the veteran. Some Veterans do not support direct questioning about firearm ownership due to fears of having firearms taken away or having their access to firearms limited. Some said they would not feel comfortable telling the truth. Veterans feel there is an opportunity to link FSS discussions with VHA's Whole Health approach. Primary care teams should use a conversational approach to FSS rather than using a script or checklist to engage veterans in a non-judgmental manner. Primary care teams should provide rationale for FSS discussions and information regarding legal consequences for disclosing firearm ownership. Providers should be transparent in their purpose for asking about firearms. Providers should acknowledge and respect veterans’ unique relationship with firearms.	Firearms storage safety (FSS) discussions are within the scope of primary care. A team-based approach to FSS in Primary Care involving nurses or veteran peers would be effective in minimizing impact on primary care clinician workflow. Directly asking veterans about firearm ownership as a way of opening up the conversation may not be the best strategy due to fear of the consequences of disclosure. Providing patient preventive or safety information, such as using Whole Health model, as context for firearm ownership questions may increase patients’ comfort levels. Primary care staff should examine personal biases about firearm ownership and prevent biases from impacting the therapeutic relationship. Primary care staff are often uncertain about legal issues including temporary transfer laws, extreme risk protection orders, and if there are circumstances when a patient's firearms might be removed or reported.
Advice on Training Content from Veterans and Primary Care Staff
The following advice/suggestions were reported in both veteran and primary care staff sessions as being important when engaging in FSS discussions:
Conveying care about patients’ safety—safety drives the discussion. Tailoring messages to the person and the situation. Expressing that veterans have control over safe storage practices. Avoiding judgmental language. Engaging in personalized and effective communication.
Note: Primary care staff also recommended training on firearm basics, having written materials such as brochures to provide more information and help normalize FSS conversations, greater understanding of firearms laws, and having scripts to facilitate firearms discussions.

From this prior work, we learned that primary care staff would like to have examples of how to effectively speak about FSS with veterans. To this end, we extracted sample messages regarding FSS by reviewing existing messaging on FSS publicly available on-line. A set of messages were presented to veterans and clinicians during individual interviews to refine a set of sample scripts for staff to be able to use during visits. To identify potential messages, extensive searches were conducted in several databases, including PubMed, PsycINFO, ISI Web of Knowledge, and The Cochrane Library for the time period from the early 2000's to 2017. In addition, we evaluated content and artifacts (eg, brochures) that had been designed to limit access to lethal means available from organizations, campaigns, and other initiatives on lethal means safety.

Specifically, we evaluated materials developed for individual firearm owners and clinicians who may interact with patients who own firearms. We identified an initial set of 60 messages, scripts, or text from awareness campaigns for our dataset. We then grouped the messages in an effort to inform our ultimate aim, that is, development of scripts for use by primary care staff to facilitate FSS conversations. After several iterations of review (by KC and EK), messages were classified as being: 1) ice-breakers (eg, opening statements), 2) context (eg, why conversation is happening), 3) facts about firearms/suicide/risk, 4) safety tips, and 5) frequently asked questions (eg, will my mental health diagnosis prevent me from owning a firearm?).

Table 2 shows the messages we abstracted from our search and presented to veterans and clinicians, and the rating scales associated with each domain. Interviews lasted for approximately one-hour and were recorded and professionally transcribed. Within each domain, we first asked participants to rate each message using Likert-type scales, and asked them to explain why they thought a particular message was acceptable or not acceptable. Across veteran and clinician interviews, we calculated mean scores of ratings (because the sample size was small, these ratings served as non-parametric indicators of acceptability). We then used a descriptive qualitative analysis approach[29-31] to identify a set of highly acceptable messages based on the expertise and experiences of the interviewee participants. After incorporating actionable qualitative suggestions (eg, wording changes or content changes), the study team combined highest rated messages in each domain to create a set of scripts organized by level of risk (ie, low risk for suicide vs. higher level of risk). For example, one script developed for lower risk patients was as follows, “Because rates of suicide by firearms are high among veterans, and depression and PTSD increase risk for suicide, I am talking to all of my patients who may have depression or PTSD about the safe storage of firearms in the home. Would it be okay if we talked about that for a few minutes?”

Table 2.

Message ranking exercise (based on means safety messages from public sources.

DOMAIN	SET-UP	SAMPLE MESSAGES PRESENTED TO STAKEHOLDERS (LISTED IN ORDER OF PREFERENCE TO VETERANS, WITH MOST STRONGLY PREFERRED MESSAFGE LISTED FIRST WITHIN EACH DOMAIN)
Ice-breakers	Patient vignette: Joe is a 56-year old veteran who is seeing his VA primary care clinician for the first time. As part of his new patient intake, Joe answers questions about PTSD and depression. Joe's depression score is high. After Joe describes his symptoms and the clinician acknowledges how he is doing, the provider can open up a conversation about means safety in several ways:	a. I do want to mention that sometimes a crisis hits and people who are already struggling suddenly experience suicidal feelings. Those feelings often go away in a matter of hours or days, but they can feel overwhelming.b. Hey, I know a lot of veterans own firearms. I don't care if you do or don't, but here are some things to think about or do in case you hit a bad place down the roadc. At the VA, we are committed to the mental and physical health and safety of our veterans and the people they love.d. Our mission is to treat patients “whole health,” both body and mind, and that includes thinking about the safety of our home.e. At the VA, we are trying to improve the accessibility and variety of mental health services and screening because suicide by a firearm is the leading cause of veterans’ deaths.f. Everyone experiences tough times. During such times, some of us may not be in the right state of mind to be handling weapons.
Why is the conversation happening?	From our meetings with veterans, we learned that veterans feel comfortable talking about weapons safety when they understand the reason for doing so (eg, patient safety). I would like to read you some sample reasons that providers could give and ask you to rate how acceptable the statements are to you.	g. As your provider, my primary concern is about your personal safety. For this reason, I want to tell you what we know about PTSD or depression and suicide.h. Some of my patients have firearms at home, and some firearm owners who are going through tough times choose to make their firearms less accessible. Are you interested in talking about that?i. Because the rates of suicide by firearm are increasing, I am talking to all of my patients who are dealing with depression and PTSD about firearm and medication safety.j. I am talking with you about all of this because, as your provider, my concern is with your health and safety.k. We know that many suicides are impulsive; and evidence shows that if you can put some time and distance between suicidal thoughts and grabbing your weapon this increases the likelihood of staying safe.l. Firearms in the home are like any other potentially dangerous household risk, such as chemicals in cleaning supplies, backyard pools, alcohol and cigarettes, prescription medication, or fire hazards. With any of these potential hazards, you can take steps to protect yourself and your family.
Facts about suicide and increased risk	Now, I would like to review some facts about suicide that providers could share with patients to underscore the importance of depression, PTSD, and increased suicide risk among veterans. I will ask you to rate the importance of each message.	m. Putting time and distance between a suicidal person and a firearm may save a life. The odds of survival go up for three reasons: 1. Suicidal crisis is often brief. 2. Deadliness of an attempt often depends in part on the method. 3. 90% of those who attempt suicide and survive don't go on to die by suicide.n. Often, people with PTSD or depression have moments where something happens and they have a crisis, and may be feeling out of control and suicidal. Studies are showing that within a 5-minute period of feeling suicidal, people may do something they might not otherwise do.o. Many suicide attempts occur with little planning during a short-term crisis. When someone feels overwhelmed in crisis, impulsive actions could tragically mean a life lost to suicide.p. Suicide using a firearm is a leading cause of death among Veterans, but it is preventable.q. Approximately 70% of veteran suicide deaths involve the use of firearms. Having access to a firearm during a suicidal crisis increases the odds of a lethal suicide attempt.
Safety tips	Now, I would like to review some safety tips that providers could share with patients to talk about weapons safety specifically. Please rate the importance of these safety tips.	r. What some veterans in your situation do is to store their firearms away from home until they are feeling better, or they use a gun lock.s. If someone is at risk, help keep firearms from them until they recover. It's like holding on to a friend's keys when they are drunk.t. Remember, nearly all firearms accidents in the home can be prevented simply by making sure that firearms are kept unloaded and locked up when not in use, with ammunition secured in a separate location.u. I would encourage you to add some barriers between you and your weapon, whether that be removing a firing pin, or handing your keys to a spouse. Don't do something you can't take back.
Frequently asked questions	Veterans have told us that one strategy to make veterans more comfortable talking about weapons safety would be to provide patients information on “VA's policy” for mental health issues- describe what will happen to a patient's weapon if she/he is diagnosed with a mental health issue, and dispel myths about weapons seizures.	v. Just having a mental health diagnosis will not cause you to lose your right to own firearms.w. In this clinic, we respect your right to own weapons, and we want to make sure that we support you.x. The VA does not have a “can't get a firearm list” for people who have PTSD or depression.y. For people who have PTSD or depression, the VA does not report firearms ownership to federal government or state officials.z. Unless we were concerned about you being an imminent risk to yourself or others, which would result in you being hospitalized, the VA would not report your gun ownership to any authorities—there would be no impact on your ability to own firearms.
Message Sources	Department of Veterans Affairs; National Shooting Sports Foundation; Project ChildSafe (a program of the NSSF); American Foundation for Suicide Prevention; Veterans Crisis Line; Rocky Mountain Mental Illness Research, Education and Clinical Center for Suicide Prevention; Defense Suicide Prevention Office; Massachusetts Office of the Attorney General & Massachusetts Medical Society; Oregon Suicide Prevention (Keys); Seattle & King County Public Health Department (LOK-IT-UP); The Suicide Proofing Initiative (Oakland County)

In addition to developing sample scripts for staff to use to guide firearms safety discussions, as was done in the initial analysis of staff and veteran attitudes,[25-27] we used a grounded theory approach[32-34] to identify candidate learning objectives for the training program and to guide program format.

We learned from focus group meetings that primary care teams and leadership preferred that training sessions be brief (preferably a maximum of 1.5 hours), as primary care staff frequently participate in trainings on competing high priority topics. Primary care teams had also expressed interest in having written materials available to help educate veterans and to normalize conversations about safe firearm storage. Suggestions included having a pocket-card which contained scripts they could examine prior to FSS discussions and patient-facing posters or brochures, which might be placed in waiting rooms or exam rooms. We therefore collaborated with VA's Office of Mental Health and Suicide Prevention (funder of this project) and obtained input from the National Shooting Sports Foundation (https://www.nssf.org/) to create a pocket card for staff and a brochure for primary care patients.

Piloting of Program

Based on information from the focus groups and individual interviews, our team developed the interactive education session, “‘Just in Case’: Discussing Means Safety with Veterans at Elevated Risk for Suicide.” The program was designed to last 1.5 hours, include approximately 20–25 participants, and to be led by two local facilitators: a mental health clinician (in this case, a psychiatrist), and a primary care provider (in this case, an internist). We also recommend including a veteran firearm owner (in our case, a VHA employee) to offer perspective regarding firearms ownership and answer questions that might regard on how firearms work. Program knowledge and skills objectives and content are presented in Table 3. The PowerPoint® presentation (available on request from the authors) includes several video segments that serve as examples and are designed to prompt further discussion. The program is broken down into approximately 50 minutes of didactive and 20 minutes of skills practice. We developed the GROW Model to provide a framework to guide clinical team members in preparing for and structuring FSS discussions (Get ready; provide a Reason for the discussion; Offer brief advice; and communicate We’re here to help).

Table 3.

“‘Just in Case’: Discussing means safety with veterans at elevated risk for suicide” program summary.

Knowledge Objectives

✓ Describe the rationale for discussing means safety with veterans in primary care. ✓ State outcomes resulting from discussing means safety with patients. ✓ Outline key steps in assessing suicide risk in general in primary care. ✓ Describe a simple model for discussing means safety with veterans. ✓ Explain how gun locks work, and specify several ways to secure firearms. ✓ Describe how patients and veterans feel about discussing firearms in the context of primary care visits. ✓ Explain what VA does or doesn't do legally with regard to restricting access to firearms. ✓ State approaches to enhancing safety for other types of lethal means (eg medications). ✓ List several approaches to initiating means safety discussions. ✓ Identify biases you may have about firearms or veterans who own firearms.

Skills Objectives

✓ Use patient-centered interviewing techniques to create rapport and facilitate open discussion about means safety. ✓ Choose approaches to opening means safety conversations that fit with your knowledge of the patient and the clinical context. ✓ Use veteran-centric language to talk about firearms and means safety options. ✓ Work with your local team to identify opportunities and roles related to facilitating means safety discussions with veterans at risk. ✓ Efficiently integrate means safety messaging into your workflow

Program Content

Welcome and introductions Review goals of course Case illustration What is (lethal) means safety? Key principles underlying means safety approaches Duration of Suicidal Crises Veterans, Firearms and Suicide Safe gun storage practices reduce risk of suicide Means Safety Interventions have potential to change storage practices Veterans’ Firearm Storage Methods Reasons veterans own firearms Majority of firearms owners and firearms organizations take safety seriously Summary of options to enhance safety How/why does this impact Primary Care? The Means Safety Messaging Approach How do patients/veterans feel about discussing firearms? How do veterans feel about discussing firearms? Primary Care Staff views Legal Concerns—VA rules/regulations Having the Conversation--When? Having the Conversation—How? Preferred Language/Terms The GROW Model and Examples Practice Session Two scenarios Pair up and trade roles (patient and staff) Group Discussion/Question and answer Useful resources Next Steps for Teams

Program materials provided in a binder for participants at the beginning of sessions included slide outlines, clinician pocket card, patient brochure, copies of several key references,[7,37] suicide risk stratification table developed by the VA Rocky Mountain Mental Illness Research, Education and Clinical Center (MIRECC) for Suicide Prevention, a brief compendium of how firearms work and firearms terminology, and a set of sample scripts to use in the skills portion of the education session and to guide discussions with patients (Table 4).

Table 4.

Final sample scripts for session participants to use for skills practice and for FSS discussions with veterans.

Reasons for the DiscussionSome things you can say if you assess that there is low risk for suicide: “I’m glad you’re not having thoughts about suicide, but Sometimes a crisis hits, and people can experience suicidal feelings. There are things you can do to help you remain safe if that were to happen. One of those things is to consider making changes in how you store firearms. Would it be okay if we discussed that?” “A lot of veterans own firearms, and as your doctor/nurse, I care about your safety. Here are some things you might want to consider…” “Because rates of suicide by firearms are high among veterans, and depression and PTSD can increase risk for suicide, I am talking with all my patients who have depression or PTSD about safe storage of firearms. Would it be okay if talked about that for a few minutes?” If you believe there is a higher level of risk: “Some of my patients who are firearm owners and who are going through tough times choose to make some changes in how they safely store their firearms. Would it be OK if we talked about that for a minute?” “A suicidal crisis can come on rapidly. We know that putting even a small amount of time and distance between having suicidal thoughts and a firearm can save a life. Some veterans choose to make changes in how they safely store their firearms.” Phrases you can use if there are others in the household: “As your provider, I am concerned about the health and safety of you and your family/friends…” “Please be aware that kids can be more curious than we might realize, and that it's common for teenagers to know exactly where firearms are hidden in the house. Are you aware of options for safely storing your firearms when they are not in use?” Offer Brief Advice If you believe there is low risk: “I would encourage you to use a locking device such as a gun lock, or to store your firearms in a lockbox or locked cabinet.” “Nearly all unintentional firearm injuries in the home can be prevented by making sure that firearms are kept unloaded and locked up, with ammunition stored in a separate location.” If you believe there is a higher level of risk: “Some veterans choose to store their firearms away from home until they are feeling better. Is that something you might consider?” “I would encourage you to store your firearm away from your home, or temporarily ask a friend or relative to store the firearm for you.” We’re here to help “Here is a brochure with some suggestions for what you can do to store firearms and medications more safely. We also have firearm cable locks that I can give you free of charge.” “Please contact our team if you have any questions about making your household safer.” If you are more concerned, you can add: “I am also giving you information on how to reach how to reach our clinic, or in an emergency, the Veterans Crisis Line.” “You’ve shared with me that you’ve been feeling really down for the past 6 months. I’m wondering if you’d be willing to meet with a behavioral health specialist who works on my team for a few minutes today to talk about how we can help you address these feelings…” If the veteran raises concerns about consequences of disclosing firearms ownership: “Having a depression or PTSD diagnosis does not legally prevent you from owning or using a firearm.” “[You have the right to own firearms]. I’m concerned about helping you stay safe since you’ve talked about having thoughts of suicide. Have you also been concerned?” “I don't need to know whether or not you own a firearm, but if you do, here are some things you might want to consider.” “I need a gun to protect myself” Response: “That's important, but there are a number of safe storage options. One option is a quick access lockbox. Here is a brochure that can help you decide which option is best for you” Response: The National Sports Shooting Foundation (NSSF) has lots of additional information on options and safety on their website.”

Program Evaluation

Methods

Between April and July 2019, we piloted the training program in two of the Medical Center's larger primary care clinics. Participants were asked to complete questionnaires just before, then immediately at the end of each session. Questionnaires included items to assess experiences and attitudes related to firearms and to discussing firearms during primary care visits. The post-session questionnaire also inquired about satisfaction with the educational program. Item responses were captured using 4-point Likert scales (4-point scale: excellent, good, fair, poor). T-tests and Chi square were used to compare pre-session to post-session responses.

Results

Together, 71 staff participated in the two pilot sessions held at two VHA clinics including all PACT members: licensed independent practitioners (physicians and nurse practitioners), registered nurses, licensed practical nurses, and medical staff assistants (MSAs). Forty-five percent of participants identified themselves as nurses, 18% as physicians, 3% as nurse practitioners, and 32% as MSAs/other support staff. Sixty-eight percent of participants were women. Eighty-six percent of staff members reported working in direct patient care 50% of the time or more. In the baseline questionnaires, 58% of participants reported ever having discussed firearms safety with a veteran, and 44% reported having offered a firearm cable lock to a veteran (note that cable locks are provided free to veterans as part of VHA's suicide prevention strategy). Across the two clinics, immediately post-session, 88% rated the quality of the program; 94% rated the usefulness of the information; 86% rated the pace of the session; 92% rated the structure of the session; and 94% rated the usefulness of written materials as “good” or “excellent”. Also, 72% responded “very likely” or “almost certain” to recommend the program to others. Free text feedback was also gathered, including what people liked most about the training, and what could be improved. The sample scripts emerged as the most liked element of the training, and suggestions for improvement included allowing more time for the training.

Participants also reported significant improvements compared to baseline in 1) the importance they placed on speaking with patients about firearms safety (3.27 vs. 2.91; p = .01); 2) ratings of perceived ability to effectively speak with patients about firearms safety (3.26 vs. 2.72; p < .0001); and 3) level of comfort speaking with patients about firearms safety (3.07 vs. 2.73; p = .034). Participants’ ratings of the extent they know enough about firearms to discuss firearm safety with patients did not change significantly post-course to pre-course (2.91 vs. 2.62; p = .085) for the 51 (72%) participants who completed pre- and post-course surveys.

Comments on Training Program Feasibility and Potential Implementation

Despite five larger clinics being invited to participate in the pilot training program, only two initially volunteered. A main obstacle noted by several clinic leaders was that there were many training needs for which they did not have time. Clinic managers also said that 1.5 hours would likely be the most time they could devote to the program if it were delivered in one session though the content and format were developed to be delivered in approximately two hours. In addition, despite our plan to present the program to smaller groups (ideally fewer than 20) to facilitate discussion and skills practice, the clinics preferred to have large groups participate in the sessions.

Discussion

This is the first project we have identified to use an iterative, stakeholder-informed process to develop and evaluate a program to teach staff to deliver firearms safety messaging in primary care settings. The education session we developed addresses knowledge and skills learning objectives and includes video demonstrations and skills practice; supplemental materials included a pocket card, which provides sample scripts to guide conversations, and a brochure to be used in primary care clinics. We piloted the program among 71 staff members, 58% of whom reported previously discussing firearms safety with veterans. An immediate post-training evaluation showed generally high satisfaction with the program, and that participants reported significant improvements in ratings of the importance of and comfort with FSS.

We identified a number of challenges to implementing the program, including competing demands on clinical leadership to find time for sessions, and their desires to have all staff participate at once. We had initially planned to deliver content in smaller group sessions to provide more time and support for skills development. We also confirmed that 1.5 hours was not enough time to cover all the material. Although one possibility would be to try to arrange for longer sessions, there may be value in separating the program into several sessions to allow for more question and answer time and post-session planning by care teams. We know from prior work that impacts of clinician training interventions can decay over time. Delivering the program over several sessions or providing follow-up interactions might also help reinforce the material and any learned behaviors.

Research on messaging models such as the SBIRT (Screening, Brief Intervention, Referral and Treatment) for risky drinking,[40,41] the “5 As” for smoking cessation, and motivational interviewing[43,44] suggest that when clinicians are trained to deliver brief sets of health messages, it can have positive impacts on patient behaviors. We found that, after delivery of this stakeholder-informed educational content, participants reported high satisfaction and an increase in comfort related to FSS. We note that in November 2020, VHA implemented a national mandatory video training session for clinical providers including primary care clinicians. While we see this as an important step and useful resource, based on the knowledge and skills gaps we learned about during our preliminary work with clinical teams, we sought to create a more intensive training experience that included knowledge and skills components. Future research is needed to assess whether the participant-reported gains are sustained over time, and importantly, whether they 1) translate to changes in practice at the provider or clinic level, and 2) ultimately impact patient behaviors and suicide outcomes.

Next steps might include delivering a larger series of training sessions and evaluating clinician behaviors (both self-reported and observed) over time. Using standardized patients to gauge staff performance or identify a cohort of patients who are recipients of FSS messaging to evaluate their responses to the messaging could be helpful. It would also be important to ensure that training components align with national consensus guidelines for firearm injury education for medical professionals as well as ongoing and increasing efforts within VHA to promote lethal means safety.

We utilized a stakeholder engagement to develop this course. We believe this approach is important given the potential sensitives of discussing firearm safety, particularly among veterans, and that including firearm owners with specialized knowledge in the process may help inform firearm storage safety recommendations. This approach could also potentially improve the credibility of course content and messengers, and allay concerns about the privacy implications of disclosing firearm access in healthcare settings. In support of this, a prior study among a national sample of firearm owners found that, among 14 different potential messengers (eg, friends, law enforcement, hunting or outdoors magazines), military veterans were ranked as the third best messengers to teach firearm owners about safe firearm storage for the purposes of suicide prevention. Notably, in this study, “physicians or medical professionals” were ranked near the bottom. Importantly, this study assessed perceptions regarding which groups would be ‘good’ at counseling, and not whether clinician-delivered counseling was acceptable or appropriate. Multiple prior studies have clearly demonstrated that firearm owners, including veteran firearm owners, agree that clinicians have a role in firearm safety counseling among at-risk individuals.[21,26,27,47]

In addition to lack of information on clinician and veteran behaviors, there are other limitations to this work. Most of the data used to develop the course were derived from individuals living in urban or suburban areas in the Pacific Northwest. Veterans and clinicians from other parts of the country or rural areas may have differing perspectives; some of the scripts we developed may not be as acceptable or effective in other patient populations. While the course was designed for care teams working with Veterans, we expect that many of the messages and approaches would be applicable or adaptable for use with non-Veteran patient populations. It is possible that some of the clinicians and nurses who did not complete post-session evaluations were less satisfied with the course.

Conclusions

This education program shows promise as an approach toward helping move suicide prevention upstream into settings that often encounter patients at greater suicide risk and who may not interact with specialty mental health. More research is needed to refine and evaluate impacts of this or similar training programs over time.