Implementation of the Short-Term Assessment of Risk and Treatability over two phases.

BACKGROUND: Over the last decade, the Short-Term Assessment of Risk and Treatability (START) has provided a strong evidence base to predict a range of problem behaviors. The implementation of START and adaptation of the services to the use of START have so far been sparsely described in the literature. The purpose of this study was to describe the continuation and the interdisciplinarity of risk assessments through the two phases.
METHODS: Over a period of 10 years, the forensic mental health services at Brøset has implemented START in two phases: initially with implementing the instrument (2005-2009) and secondarily by customizing the instrument to everyday treatment and planning (since 2009). This implementation was based on data from 887 START assessments for 181 patients over a decade (2005-2015).
RESULTS: The results showed that the number of START assessments has been stable throughout the past 10 years and the interval between the ratings has decreased significantly (p<0.05). The involvement by diversity of professionals has increased significantly over the two implementation phases.
CONCLUSION: This study also addressed the continuity and organization of the implementation process and presented an overview of how START has been widespread in the service through treatment. The results showed an increased multidisciplinary participation and a continuing rate of assessments as the implementation progressed from assessment to a combined assessment-treatment phase.

Introduction

Risk analysis has gained increasing importance for mental health nurses working in secure/forensic inpatient services.1 During the last 20–30 years, research and development of various schemes, tools and instruments have been carried out to help health care providers to better work with high violence risk.2,3 Such tools are meant to incorporate not only static/historical factors but also dynamic factors relating to the individual’s risk for violence against others.4 Despite their popularity, structured judgment schemes have been criticized for their strong focus on the presence of factors that are associated with an increased risk, while ignoring the presence of protective factors that may diminish risk.5 Although clinicians have sought to bring out the positive characteristics in their patients and aimed to provide external support, the introduction of positive treatment models such as the Good Lives Model6 has recently been more explicitly incorporated into treatment.7

The Short-Term Assessment of Risk and Treatability (START)3,8 is a structured clinical judgment instrument that aims to support the assessment of patients with mental illness for a diversity of adverse outcomes through the assessment of dynamic risk and protective factors (termed as Vulnerabilities and Strengths). The tool was developed in Canada and has been adapted and validated to services in the UK, Australia, the Netherlands, Ireland and Norway.9 START is a clinical assessment focusing on dynamic variables, including vulnerabilities and possible strengths. It aims to help mental health professionals with structuring the risk assessment processes in practice and organize the health care services to reduce risk and increase strengths in the patient. Importantly, START aims to guide risk formulation for a range of adverse outcomes that are faced by people with mental disorders in addition to violence, namely, self-harm, suicide, self-neglect, victimization, substance abuse and unauthorized leave.9 Moreover, information from START assessments may be useful by facilitating the discussions, decision-making and risk communication in the treatment team.3,10 START ratings have demonstrated a high internal consistency, inter-rater reliability and convergent validity with other risk measures.9 In this meta-analysis, there was limited information about the variability of START ratings over time. Inter-rater reliability and convergent validity with other established assessments of risk and protective factors have been found to be good, and studies have shown a high acceptance by mental health professionals.9 In a Norwegian study,11 results indicated that START was a valid violence risk assessment tool. Risk estimates of START have shown strong predictive validity for various aggressive outcomes and good predictive validity for self-harm9 and ultra short-term assessments of violence risk such as the Dynamic Appraisal of Situational Aggression (DASA)12 and the Brøset Violence Checklist (BVC),13 which assess risk for the subsequent day. No rationale is offered in the START manual8 for deciding a 3-month inter-assessment period. Studies about the predictive validity of START have used different follow-up periods varying from 30 days to 1 year,9 but there are limited data to determine whether the length of a follow-up period moderates the effect size of predictive outcomes. One study found the START Vulnerabilities scale to be a significant predictor of aggressive outcomes at periods of 1, 3, and 6 months using receiver operating characteristic (ROC) curve analysis (area under the curve [AUC] range 0.74–0.83) but did not examine the tool’s Strengths scale and its specific risk estimate (SRE) or nonaggressive outcomes.14 Wilson et al15 have reported statistically significant AUC values for the Strengths scale, Vulnerabilities scale and SRE for violence at 3-, 6-, 9- and 12-month periods following assessment; again, these studies only examined aggressive and sexually inappropriate behaviors and not the full range of START outcomes. This is an important question since more accurate prediction over one period of time compared with another would have implications for the frequency with which reassessment is conducted. This information is useful since risk management procedures that involve restrictive practices should be adapted to the individual’s level of risk at any given time. Furthermore, reassessment at too short intervals may impair valuable resources.16 It is important to study the period following assessment in which patients do not expose risk outcomes (the so-called survival period) since such engagement might in itself secure reassessment. In clinical mental health services, it is important to remember that the goal of risk assessment is not simply to predict the likelihood of violent behavior but to reduce its risk. Therefore, the risk assessment tool must be implemented in a sustainable manner, findings from the risk assessment tool must be communicated accurately and completely and information derived from the risk assessment process must be used to support risk management and rehabilitation.17 According to Otto and Douglas,5 the new challenge for structured clinical risk assessment instruments is to ensure fidelity of application. Design of a risk assessment instrument may be easier than ensuring its true-to-purpose application in forensic, civil mental health and correctional settings.18

Implementation over two phases

Implementation refers to the process of putting a procedure into practice.19 The implementation of assessment tools is primarily focused on evidence-based tools that have been developed and validated. No matter how precise the assessment tool is, it will fail if it is not implemented correctly or used consistently.20 The implementation of risk assessment tools as a continuum gives a large number of challenges as described by Nonstad and Webster.18 In summary, a validated tool for assessing violence risk relies on the implementation and the value of assessing risk depends upon several factors beyond the quality of the tool.

Implementation of research comprises the study of processes and strategies that integrate evidence-based treatments into routine use in care settings. Understanding these processes is crucial for improving care, and systematic, empirical research on implementation is needed.19 There is considerable evidence-based literature for the usefulness of structured clinical instruments for assessing risk in the mental health services. In stark contrast to this evidence, there is sparse research on methods of implementing these instruments. Research on implementation requires outcomes that are conceptually and empirically distinct from those measuring the validity of an assessment. These include the assessments’ penetration within an organization, its acceptability to and adoption by different stakeholders, the feasibility of its use and its sustainability over time within the service system setting.19

A process of implementation of risk assessment tools has been suggested by Vincent et al,20 following eight steps from a preparation phase until the final step of making the risk assessment practice sustainable. Implementation is further divided into two time periods: 1) pre-implementation – after staff received training on a risk assessment tool but prior to implementation of a clear office policy training about how to use the tool in decision-making, and 2) post-implementation – after office policies and training on the use of the tool in decision-making occurred and were applied in practice.

The preparation of getting ready to implement involves numerous stakeholders both at the administrative and clinical levels. The first step in the process of implementing START at the clinic in St. Olav’s Hospital, Brøset, was to introduce and describe START to the staff. The aim was to obtain trust among the staff and explain the usefulness of using a structured professional judgment tool in the everyday clinical setting. Through a general introduction of START and many informal discussions/meetings, the majority of the staff realized the opportunities that the instrument had concerning risk assessment, risk management, multidisciplinary cooperation and planning. The ideas of START were developed in the ward between the START group and the staff. After START was presented to the staff, the instrument was introduced and described to the management of the hospital. The aim was to have the necessary support in the organization when implementing the tool.

At the further steps of a suggested implementation, establishing stakeholders, preparing the tool, preparing a policy for decision-making, training, pilot testing, full implementation and ongoing tasks for sustainability, the service organized a dedicated START group put together by a multidisciplinary professional group. A system of monitoring, educational support, supervision and research was formally established, as well as routines twice a year reports. A survey of the staff confidence in using the START showed that between 75 and 90% of the staff found START useful or needed as a professional tool.21 National White papers have demanded violence risk assessments including START as a recommended tool in mental health specialist services. START was familiar in the universities and services of other regions, and neighboring countries have received training from members of the START group.

While START has been used internationally and there is some evidence for its feasibility, utility and predictive value, the implementation and maintenance in the application of the tool have not been investigated. The current study aimed to describe the continuation and the multidisciplinarity of risk assessments through two phases: 1) structuring the use of START (2005–2009), and 2) combining risk assessment with treatment planning (since 2009). The compliance of assessments was studied based on the rating frequency and professional involvement. A brief comparison between assessment practices for short-term and long-term patients was conducted as we hypothesized a higher rating frequency in short-term patients.

Methods

Setting and implementation of START

St. Olav’s Hospital, Brøset, serves regions in the central and northern parts of Norway with 1.2 million inhabitants.

This implementation study of START was performed in 2015 based on data from 887 START assessments for 181 patients over a decade (2005–2015). The study was carried out at the Regional High Secure Hospital Brøset in Trondheim, Norway. The unit has 18 beds, and most of the patients are short-term patients referred from general psychiatric care units because of severe mental disorders combined with severe behavioral problems, mainly of a violent nature. Since the introduction of START, a total of 99 separate nurses have been involved in the ratings. New staff members have been educated and engaged in the ratings throughout the years. A majority of the staff have had formal training in risk assessment. Approximately 30 patients are admitted per year, mostly on an involuntary basis. Nursing staff on the wards constitute 78.5 full time equivalents (FTEs), of which 33 are psychiatric nurses. The units are staffed with 2.5 FTE psychologists, 2.5 FTE psychiatrists, and 2.5 occupational therapists (OTs). The admission period varies from a few days to one patient staying for the whole registered period (median=91 days). Figure 1 illustrates the distribution.

Figure 1

The length of admissions at the security forensic mental health unit at Brøset (n=174).

Integrated within the hospital is a research and educational unit that has responsibilities to both perform research affiliated with the Norwegian University of Science and Technology, Trondheim, Norway, and teach duties to the forensic psychiatric units, low-security units and prison system in the region.

Procedure

Treatment plans, security measurements and START were collected on a regular basis. According to the assessment protocol of Brøset, the first START should be accomplished within 3 weeks after admission. Later, a START assessment should be made every second month. Treatment planning and security measurements are independently written and adjusted throughout the admission. All data were organized by the first author who has the mandate of being the nursing supervisor of risk management at the hospital. The first author also loaded the data and prepared it for analysis.

Measurements

START comprises 20 dynamic items, each scored by a multidisciplinary team on two 3-point scales in terms of both risk factors (Vulnerabilities) and protective factors (Strengths): 0 indicates no/minimal vulnerability or strength evident, 1 indicates moderate vulnerability/strength, and 2 indicates high vulnerability/strength. The teams of raters then make a risk evaluation (low, moderate or high) about the likelihood of each of seven risk outcomes occurring over the next 3 months: violence to others, self-harm, suicide, substance abuse, victimization, self-neglect and unauthorized leave. All START ratings are dated, and the multidisciplinarity of rating teams is described.

Treatment planning is based on the terminology of START and is adapted to be evaluated by the consecutive START assessments. The plans serve as clinical guidelines to the staff.

Leveling security on a scale from 0 to 7 is a measure of considering the treatment outcomes throughout the admission period. All patients are set on level 0 when they are admitted, and the levels are dynamically evaluated. At level 0, the patient has very limited access to self-determined activities, and at level 7, there are less limitations and the care mostly concerns necessary agreements between the hospital regime and the patient. These levels were rated independently from the risk assessments.

Data analysis

All data were loaded into the Statistical Package for the Social Sciences (SPSS) version 19 and analyzed by the means of statistical methods. Demographic analysis provided information about the length of admissions. Chi-square tests were used to compare the multidisciplinarity of START assessment over two time periods, and analysis of variance (ANOVA) was used to study the persistence of START ratings and comparisons between short- and long-term patients.

Ethical considerations

The study was based on clinical data, and an application for ethical approval was made. The Norwegian Regional Committee for Medical and Health Research Ethics (central region) described the study as quality assurance and evaluation, which does not require approval.

Results

One of the aims in implementing START was to make it useful as a multidisciplinary tool. The multidisciplinary teams consisted of psychiatric nurses, OTs, psychiatrists and psychologists, of whom the psychiatric nurses have been involved in all START ratings. The participation of the other professionals was studied for the two time periods 2005–2009 and 2009–2015 (Table 1).

Table 1

The multidisciplinarity of START assessments over two time periods: 2005–2009 (Period 1) and 2009–2015 (Period 2)

Professional groups	Period 1	Period 2	Chi-square test
Number of START assessments	413	474
OT, n (%)	74 (17.9)	122 (25.7)	χ2=7.84, p=0.006
Psychiatrist (MD) and/or psychologist, n (%)	270 (65.4)	335 (70.7)	χ2=2.86, p=0.1
OT and MD and/or psychologist, n (%)	55 (13.3)	98 (20.7)	χ2=8.37, p=0.004
OT or MD and/or psychologist, n (%)	288 (69.7)	374 (77)	χ2=5.99, p=0.015

Abbreviations: START, Short-Term Assessment of Risk and Treatability; OT, occupational therapist; MD, medical doctor.

Persistence

To implement START, regular training was considered as the main target. All patients were assessed within 2–3 weeks after their admission. The interval between the START ratings was further studied. An overview of the total number of ratings is given in Table 2. The time intervals between START ratings were studied and compared between the two periods. The mean interval through period 1 (n=315) was 56.45 days (SD=32.6), and at period 2 (n=375), it was 51.99 days (SD=23.9). An independent t-test showed a significant difference in the time interval between the two periods (t=2.07; p=0.04).

Table 2

The number of START ratings over the period 2005–2015

Year	START (n)	Patients (n)	Number of ratings per patient
2005	81	27	3.0
2006	106	38	2.8
2007	75	23	3.3
2008	86	31	2.8
2009	59	25a	3.2
2009b	21
2010	68	22	3.1
2011	94	26	3.6
2012	83	23	3.6
2013	85	25	3.4
2014	64	20	3.2
2015	50c	15	3.3

Notes:

All patients from 2009 were studied as one group.

Number of START ratings after October 2009.

Data were collected until September 2015.

Abbreviation: START, Short-Term Assessment of Risk and Treatability.

Long-term patients

The patients of the forensic unit at Brøset were admitted for both short- and long-term periods. The long-term patients were convicted for serious offenses, primarily homicide. We studied the interval of START ratings and compared long-term patients (admitted for >1 year) with short-term patients. A total of 38 patients were admitted for >1 year, giving an interval of START ratings with a mean of 57.9 days (SD=28.4). Short-term patients (n=144) were rated with intervals of 43.8 days (SD=21.5). The difference in intervals of START ratings between short- and long-term patients was significant using an independent t-test (t=6.2; p<0.001).

Discussion

START was implemented over the last 11 years at the Regional Secure Mental Health Services, Brøset. From the results, it was found that the assessment tool has involved an increasing multidisciplinary team and the treatment is today based on professional evaluations from START. To enable the services to benefit from this tool, there has been an objective of keeping a high frequency of ratings, and as most patients are defined as short-term patients (admitted for <1 year), the monitoring effect has been of increased relevance. The results showed that the number of START assessments has been stable throughout the past 10 years and the interval between the ratings has even decreased. According to the manual, START is suited to a 3-month interval of assessment, but as most of the present patients are admitted for short periods, the interval is also shorter. For some patients, the only assessments are at the time of admission and the time of leaving the ward. For others, the progress through the hospitalization needs to be monitored as treatment plans are outdated.

Considerations about the implementation have been the main objective of this article, and there are several factors that may have influenced the way START is working at the services. In concordance to the phases and steps described by Vincent et al,20 there was a shift in 2009 when the implementation of the risk assessment was connected to the treatment planning and integrated into the electronic patient journal system. The content of the steps suggested by Vincent et al20 was followed, but the initial steps were more interwoven. At the initial phase promoting START, education, consulting stakeholders, translation and adaptation into Norwegian language and establishing routines for application were parallel priorities. Prior to this change, all stakeholders were involved and became confident with using START.

A large number of factors may have added to the success of the present implementation process. First, the staff were involved at an early stage of implementation, with some of the staff members as START supervisors. A broad educational program was given to all staff members independent of their professional position. All new employees at the service were educated in the principles of risk assessment in general and specifically with START. Second, the infrastructure of a few experienced supervisors and experts within the assessment and management of risk with a multidisciplinary professional background had served as both internal and external promoters for the continuing efforts. The certainty that good practice depends upon continuing processes with shared knowledge had been stressed during the whole implementation period.

Many services have initiated risk assessments based on mandates from the health authorities or collaborating parts, and numerous attempts have failed. Such failure may depend on several mistakes as described by Nonstad and Webster.18 However, a unit with a high degree of patient turnover, as there is in the studied population, calls for a higher intensity of assessment. Through the quantity of training with medium-to short-term admissions, the staff will be both experienced assessors and able to document changes during treatment.

Limitations

Risk assessment and management using a single assessment tool (START) may be too general for certain patients or patient groups, i.e., sexual offenders or patients with learning disabilities. On the other hand, the treatment culture and professional identity which evolved by implementing START benefit patients who might have been treated evidentially better. The implementation of START is a continuing effort at the present service where two challenges seem most prominent: 1) making staff more confident in transferring the assessments into treatment plans, and 2) creating a continuity of the START principles when the patients are leaving the secure mental health services and later being treated at a less restrictive level of the mental health services. This paper has not presented the patient outcomes based on the introduction of the treatment plans and electronic journal. However, these data are available and will be presented in a future paper.

Conclusion

The implementation of START has been studied as a result of the maintained intensity of START ratings and the multidisciplinary participation. Over two periods of this 10-year implementation, the multidisciplinary participation has increased and the frequency of the assessments has been maintained at 3–4 months.