Desperately seeking outcomes: quantifying the effectiveness of community mental healthcare using Health of the Nation Outcome Scales.

BACKGROUND: Community mental health services in the UK have struggled to measure the clinical effectiveness of their services.AimsTo measure clinical outcomes for different diagnostic clusters.
METHOD: Clinicians measure the clinical status of patients by the Health of the Nation Outcome Scales (HoNOS), and HoNOS scores should be recorded annually after treatment. Clinical outcomes were measured by changes in HoNOS for diagnostic clusters.
RESULTS: In two time periods (2014 and 2016), the health of patients with mild to moderate common mental disorders deteriorated after intervention. Patients with severe common mental disorders and psychoses improved in their clinical status.
CONCLUSIONS: British community mental health teams may be effective in improving the clinical status of people with severe mental illness, but may have a negative effect on people with mild to moderate illnesses. These teams need to focus on the severely mentally ill and build on this demonstrable effectiveness.Declaration of interestNone.

In most healthcare arenas, outcome data are key in measuring the quality of healthcare. In some branches of medicine this can be done easily, for example by measuring infection or death rates after surgery or survival rates in cancer or following a stroke. This has proved more difficult in mental health, where suicide/homicide rates are too multi-factorial and (thankfully) rare, and patient satisfaction is an unreliable measure for evaluating clinical outcomes.

The Health of the Nation Outcome Scales

The Health of the Nation Outcome Scales (HoNOS) were developed by the Royal College of Psychiatrists in the 1990s, at the request of the Department of Health, with the aim of measuring health and social functioning of people with mental illness. It is comprised of 12 scales covering symptoms, functioning, relationships and environmental issues. Each domain is graded by the treating clinician from zero to four: where zero means no problem, one means there is a problem probably requiring intervention, and two, three and four correspond to ‘mild’, ‘moderate’ and ‘severe’ problems. A score of nine is allocated if the clinical state is unknown. Currently, HoNOS is the most widely used clinical outcomes tool used by psychiatric services in England. This is partly due to the development of a commissioning tariff based on a Mental Health Clustering Tool, which requires HoNOS scoring for all patients within the scope of the mental health tariff.

The Mental Health Clustering Tool includes HoNOS with some additional scales and is used to group patients into clusters that are broadly diagnostic (see Appendix). Clusters 1–5 represent a single non-psychotic episode, 6–8 represent an enduring non-psychotic illness and clusters 10–15 represent variants of psychotic illnesses. Despite using HoNOS, very few centres have reported results from this data source.

Validity of HoNOS

Various studies in the past 20 years have examined the validity and reliability of HoNOS as an outcomes tool. Orrell et al showed HoNOS to have good validity but variable reliability. Brook et al, looking at three studies undertaken in Australia, highlighted that the inter-rater reliabilities were at best moderate and the validity does not correlate with any major measure of mental health symptoms, nor with a major measure of health status. However, a more positive outlook was presented in a review of the psychometric properties of the HoNOS family of measures; it was found to include reasonably good content, construct, concurrent and predictive validity. A recent study which focused on using HoNOS during the in-patient to community transition has again reiterated the positive value of HoNOS.

Context and aims of the study

Cornwall, a county in the south west of the UK, has a population of 538 000. It is a rural area with high rates of deprivation and suicide compared with other counties in England, but low rates of psychosis. There is one National Health Service (NHS) Mental Health Trust in Cornwall that provides secondary healthcare for people with a severe and enduring mental illness. Healthcare is provided both in hospital and in the community, through wards and various community mental health teams (CMHTs), and there is a strong emphasis on continuity of care. The HoNOS clustering tool should be completed when patients are taken on for care, and then annually thereafter.

We sought to ask the following research questions: which patient groups clinically improve after receiving care from secondary mental health services as measured by HoNOS? Are HoNOS data being adequately collected to help improve services?

Method

In 2014, we addressed the research questions by examining changes in individual patient HoNOS scores. Cornwall Partnership NHS Foundation Trust provides the county with secondary mental healthcare and uses an electronic data management system called RiO. The HoNOS score was part of the clustering tool (see above) and therefore at least one HoNOS score was recorded at the initial patient assessment. Staff are trained to use the clustering tool which includes training in scoring the 12 items that comprise the HoNOS scale. Training is delivered to maximise reliability and validity and is consistent with the original HoNOS training devised by the authors of the scale.

The records of all patients under the care of the Trust were examined to determine if at least two HoNOS scores had been recorded during treatment. If more than two HoNOS scores were recorded, the difference between the latest HoNOS score and that measured at the initial assessment was used.

In 2016, a similar examination was completed but with greater focus on how many patients had been measured at least twice using HoNOS. All patients under the care of adult CMHTs in Cornwall between 1 June 2015 and 30 April 2016 were included, using the same methodology as the previous 2014 audit. HoNOS scores at the point of referral to CMHTs and a second HoNOS score at a future point during the patients' care were analysed. The impact of clinical interventions was reviewed by looking at the number of patients who showed improvement, deterioration or no change in their HoNOS scores at the end of an episode of care.

We did not seek ethical approval as the research was a service evaluation and we did not use any identifiable individual patient data.

Results

The data for 896 patients in 2014 are shown in Table 1 and Fig. 1. In this sample, any unknown scores of nine were discounted as zero. When grouping patients by cluster, we observed that the HoNOS scores for patients in clusters 1–3 deteriorated and there was little change for patients in cluster 4. Patients in clusters 5–8 and 12–17 showed improvement. This suggested clinical improvements in patients with severe non-psychotic illnesses and psychosis; but less severe, non-psychotic illnesses worsened for patients with input from the CMHT. Data were used for the minority of patients with at least two HoNOS scores, but the number of patients without this data was not recorded. As a result of this data, staff were encouraged to follow the established policy of transferring care of patients in clusters 1–3 (patients with a mild to moderate non-psychotic illnesses) to the separate primary care counselling/cognitive–behavioural therapy service.

Table 1

Changes in mean Health of the Nation Outcome Scales scores by patient cluster in 2014

Initial cluster	Total clients	Average initial score	Average recent score	Increase or decrease (−) in average score*
0	1	14	14	0.00
1	5	4	10	6.00
2	15	10	14	4.60
3	79	11	14	2.63
4	143	15	15	0.06
5	40	17	14	−3.10
6	16	16	15	−1.25
7	45	17	16	−0.58
8	52	18	17	−0.92
10	25	14	13	−0.56
11	58	11	11	0.62
12	34	15	12	−3.03
13	26	18	14	−3.73
14	34	19	14	−5.12
15	13	21	15	−5.31
16	15	18	17	−1.27
17	33	18	15	−3.21
18	90	7	9	1.92
19	99	12	13	0.54
20	40	18	17	−0.98
21	33	18	19	1.15

* The average HoNOS scores are rounded up or down, but the average difference between the initial and subsequent HoNOS scores for each patient is given to the nearest 0.01 change.

Fig. 1

Changes in mean Health of the Nation Outcome Scales scores × patient cluster in 2014.

In 2016, 1579 of 6207 patients were not allocated to a cluster group at the point of referral and were thus not included in the study. All patients with incomplete HoNOS scores and those with a score of nine – which indicates unknown disability/severity of disability – were excluded from analysis. As seen in Fig. 2, there were 4409 patients with a complete HoNOS assessment recorded, which was used in the clustering process. However, more than two-thirds of patients had only one HoNOS assessment and therefore their data could not be used for comparison. As such, we could only use data from 1271 patients who had at least two complete HoNOS scores recorded.

Fig. 2

Flowchart of patients used in the 2016 study.

Patients were grouped according to their cluster allocation based on the Mental Health Clustering Tool. The average HoNOS scores at first assessment and final assessment were then compared for patients in different cluster groups. For clusters 1–4 (common mental disorders: mild to moderate), the mean HoNOS score at first assessment was 12.35, increasing (suggesting a worsening health status) to 14.47 at final assessment. In clusters 5–8 (severe common mental disorders and personality disorder), the average HoNOS score at first assessment was 17.08, decreasing to an average of 15.45 at final assessment. The average HoNOS score in clusters 10–16 (psychoses) improved from 15.2 at first assessment to 13.88 at final assessment; whereas the average score in clusters 17–21 (dementia) increased from 11 at first assessment increase to 12.5 at final assessment (a difference of 1.5), which suggests worsening functioning (Table 2, Fig. 3).

Table 2

Changes in mean Health of the Nation Outcome Scales scores by cluster groups in 2016

	Mild to moderate common mental disorders (cluster 1–4)	Severe common mental disorders and personality disorders (cluster 5–8)	Psychosis (cluster 10–16)	Dementia (cluster 17–21)
Average HoNOS score at first assessment	12.35	17.08	15.2	11
Average HoNOS score at final assessment	14.47	15.45	13.88	12.5
Average HoNOS score improvement	−2.12	1.63	1.32	−1.5

HoNOS, Health of the Nation Outcome Scales.

Fig. 3

Changes in mean Health of the Nation Outcome Scales (HoNOS) scores × cluster groups in 2016.

Discussion

There are limitations to this study. In 2016, despite 6207 patients being under the care of CMHTs in Cornwall, only 1271 (20.5%) of patients were eligible to be included in the study following the application of appropriate exclusion criteria. Although this still represents a reasonable sample size, the large percentage of patients that were not eligible to be included suggests that the change in HoNOS score is not culturally embedded as an outcome measure for clinical services.

There may be differences between the people who have two HoNOS assessments and those who do not. Those who have a second HoNOS score may be those who do not benefit from input and thus use the service for a longer period. Patients who recover may be discharged earlier, without a second HoNOS assessment. Any such differences might introduce bias in the results. Additionally, patients who have remained in the service for a longer time may be more likely to be unwell, leading clinicians to detect a greater degree of disability and therefore record an increased HoNOS score. However, this might also be true for people with clusters 5–8 and 10–16, but scores improve for these patients.

Using the score of nine to indicate unknown disability/severity of disability is problematic as it makes using HoNOS redundant since the final value does not accurately reflect the patients’ disability and its severity.

Using the change in mean HoNOS scores to compare the quality of clinical outcomes can be unhelpful as the score contains 12 variables that are indicative of different disabilities. Tracking individual scores for each of the 12 questions included in HoNOS may give a better understanding of the impact our interventions have on patients’ health and wellbeing. Other studies have examined the HoNOS in this way, but this method is expensive as more work is required to analyse the data.

This is a single-site study and as such may not be representative of other NHS services. A study looking at multiple Trusts may provide more data and identify if these results are unique to Cornwall or if this is a trend seen across multiple Trusts. We have found similar data in only one study and, although they presented their data in a very different way, they suggested people in clusters 1–5 did improve.

Although these limitations suggest caution in interpreting these results, our findings suggest that it may be counterproductive for patients with mild to moderate mental health disorders to have input from CMHTs. It is possible that CMHT care has a negative effect on people with milder mental illnesses. Staff might be better trained for treating people with more severe illness, and their interventions might worsen milder problems. It is striking that the patterns for deterioration in HoNOS scores in clusters 1–3 occurred in both cohorts from 2014 and 2016.

More encouragingly, the data demonstrate that CMHTs tend to perform better for patients with severe and complex non-psychotic and psychotic disorders. It is not surprising that patients with cognitive impairment do not tend to see an improvement following intervention from CMHTs according to HoNOS score as dementia is a progressive illness. The worsening HoNOS scores in this group gives validity to the use of the average total score.

It is surprising that HoNOS has not been used for outcome data but rather as a clustering tool in England. Mental health services have struggled to establish outcome measures for their service and have focused on process measures instead. We need to use outcome measures to establish which services are effective in improving people's lives. This can help drive the investment in useful mental health services when we are effectively competing for funds with other areas of medicine that can clearly demonstrate clinical benefit.

Although HoNOS has limitations in validity and reliability, we believe there are opportunities to use changes in total HoNOS scores to demonstrate service effectiveness and performance at all levels, from the individual patient to the team and Trust. At the moment it is the best outcome measure we have as it has been used for nearly a decade.

Our data suggest that CMHT input is effective for patients with severe and enduring mental illnesses (clusters 5–16), which is good news. However our results suggest that for mild to moderate mental illness, CMHT input is not effective. This strengthens the case for having clear boundaries in referring those patients to counselling/primary care talking therapies.

Cluster number	Cluster label
1	Common mental health problems (low severity)
2	Common mental health problems
3	Non-psychotic (moderate severity)
4	Non-psychotic (severe)
5	Non-psychotic (very severe)
6	Non-psychotic disorders of overvalued ideas
7	Enduring non-psychotic disorders (high disability)
8	Non-psychotic chaotic and challenging disorder
9	Blank cluster
10	First episode in psychosis
11	Ongoing recurrent psychosis (low symptoms)
12	Ongoing or recurrent psychosis (high disability)
13	Ongoing or recurrent psychosis (high symptom and disability)
14	Psychotic crisis
15	Severe psychotic depression
16	Dual diagnosis (substance misuse and mental illness)
17	Psychosis and affective disorder (difficult to engage)
18	Cognitive impairment (low need)
19	Cognitive impairment or dementia (moderate need)
20	Cognitive impairment or dementia (high need)
21	Cognitive impairment or dementia (high physical or engagement)