Cluster randomised controlled trial of a guided self-help mental health intervention in primary care.

OBJECTIVES: To ascertain whether an ultrabrief intervention (UBI) improves mental health outcomes for patients in general practice with mild-to-moderate mental health concerns. TRIAL
DESIGN: Two-arm cluster randomised controlled trial.
METHODS: Participants: general practitioners (GPs) were invited based on working in a participating general practice. Patients were eligible to participate if aged 18-65 years, scored ≤35 on the Kessler-10 (K10) and if meeting local mental health access criteria (based on age, low income or ethnic group).
INTERVENTIONS: intervention arm GPs were trained on the UBI approach, with participating patients receiving three structured appointments over 5 weeks. GPs randomised to practice as usual (PAU) did not receive training, and delivered support following their existing practice approaches. OUTCOME MEASURES: primary outcome was patient-level K10 score at 6 months postrecruitment.Randomisation: GP practices were randomised to UBI training or PAU at the start of the study.Blinding: GPs were not blinded to group assignment.
RESULTS: Numbers randomised: 62 GPs (recruiting 85 patients) were randomised to UBI, and 50 to PAU (recruiting 75 patients).Numbers analysed: 31 GPs recruited at least one patient in the UBI arm (70 patients analysed), and 21 GPs recruited at least one patient in the PAU arm (69 patients analysed). OUTCOME: K10 scores from an intention-to-treat analysis were similar in UBI and PAU arms, with a wide CI (mean adjusted K10 difference=1.68 points higher in UBI arm, 95% CI -1.18 to 4.55; p=0.255). Secondary outcomes were also similar in the two groups.
CONCLUSIONS: the UBI intervention did not lead to better outcomes than practice as usual, although the study had lower than planned power due to poor recruitment. The study results can still contribute to the continuing debate about brief psychological therapy options for primary care and their development. TRIAL REGISTRATION NUMBER: ACTRN12613000041752; Pre-results. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

Pragmatic effectiveness trial of a mental health intervention in primary care.

Intervention included Māori cultural adaptations.

Recruitment issues limit strength of results.

Intervention was applied to more severe mental health presentations that it was developed for.

General practitioner degree of adherence to the intervention tool is unclear.

Introduction

Mental health is major aspect of health and poor mental health is highly prevalent in the general community. Consistent with international findings, just under 40% of the New Zealand (NZ) population had met criteria for a diagnosable mental disorder during their life, and roughly a fifth had experienced a mental disorder in the previous year.1

There is also considerable international concern about the healthcare burden arising from mental health problems and substance abuse2–4), with the World Mental Health Survey (of 21 countries) suggesting that only 41% of people with depression received treatment that met even minimal standards.5

In NZ, as in other OECD countries, mental health problems are common presentations in primary care. Around one-quarter of primary care patients (26.5% and 29.8% of men and women, respectively), attending their general practice in NZ met criteria for a mild-to-moderate mental health disorder and an estimated 50%–70% of mental health concerns are managed exclusively at the primary care level, since secondary care services have become more targeted towards severe and enduring mental illness in recent years.6

Internationally, there is a call for psychological therapies to be more widely available in primary care,7 and growing unease about increasing levels of antidepressant medications being prescribed compared with the limited resources available for psychological interventions.8 However, treatment options at the primary care level are limited, with GPs expressing concerns about gaps in services for patients with mild-to-moderate mental health presentations and a desire to offer a brief intervention themselves.9 In NZ, GPs reported that as few as 22% of patients with mild-to-moderate mental health syndromes receive any formal help.10

Such patient presentations often comprise subthreshold syndromes,11 12 and cases of mild-to-moderate common mental disorder. These are combinations of problems such as anxiety, depression, substance use and interpersonal problems that do not meet the threshold for disorder in standard diagnostic systems such as Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. Often these arise in the context of social problems or family or economic stress. In NZ, 36% of general practice attendees report anxiety, depression or substance use, or a combination of these issues.6 Such presentations can be associated with significant impairment in functioning and suffering,13 14 with some going on to develop severe depression.15 16 Intervention may be warranted for up to 80% of those affected,10 13 but referral out of the practice can be problematic due to referral eligibility criteria, waiting times, administrative issues and cost.9 17 18

Increasing knowledge of the burden of mild-to-moderate disorder led to the development of a platform of Primary Mental Health Initiatives in NZ, which included some increase in access to psychological therapies and extended consultations with GPs. The inclusion criteria for these initiatives, however, mean that only up to 15% of the population can gain access to those services.9

This service-gap led us to develop a GP delivered ultrabrief intervention (UBI), with development and refinement based on service user feedback.19 This model has the advantages of avoiding the need for referral on to an expensive professional, such as a psychologist, of being easily accessible to patients and of potentially building on existing trusted relationships. This fits with the movement towards alternative methods of service delivery for mild-to-moderate mental health presentations, often termed ‘low-intensity’ interventions. These interventions often include guided self-help, bibliotherapy and computerised delivery of care, with current evidence suggesting that even minimal therapist contact leads to better outcomes than self-help alone.20–23

UBI was feasibility tested with a group of 16 patients and then adapted for Māori (the indigenous people of NZ) and feasibility tested with a group of 9 patients.24 25 Based on questionnaire feedback, clinician and patient satisfaction ratings for both feasibility studies were very positive in terms of relevance and acceptability. The psychological well-being of the patients, as measured by the Kessler-10 (K10),26 was also significantly improved postintervention (at 3-month follow-up) for both Māori and non-Māori, although there was no control group.24 25 Based on these initial findings, we designed a cluster randomised controlled trial to measure the effectiveness of UBI.

The aims of the study were to compare patient-level outcomes on (1) mental health state (as measured by K10 scores) at 6 months between UBI and practice as usual (PAU) study arms (primary outcome) and (2) levels of distress (depression and anxiety) and functioning (work, social and relationship) at 8 weeks and 3 months between UBI and PAU study arms (as secondary outcomes).

Methods

A protocol for this study has been previously published, and includes description of planned analyses.27 The trial was registered prior to recruitment commencing with the Australia New Zealand Clinical Trials Registry (registration ACTRN12613000041752.)

Design

We used a pragmatic two-arm single-blinded, cluster randomised controlled trial of UBI compared with PAU, in a primary care setting. GPs were randomised by practice to exclusively deliver either UBI or PAU to all their recruited patients. GPs were treated as the clusters in the study design (while there was clustering by practice, the GPs were treated as the unit of analysis as practitioner attributes were anticipated to be a higher source of variability in outcomes). Analysis followed an intention-to-treat approach.

Setting

The study was conducted in general practices in the greater Wellington region, NZ. This included practices in both city and semi-rural settings, serving populations from a wide range of socioeconomic backgrounds. Recruitment took place between 1 May 2013 and 1 July 2016. The trial ended prior to achieving the final sample size when funding for data collection was exhausted.

Participants

This was a pragmatic trial supported within existing treatment services. GPs were eligible to participate if they were currently working in a practice that was part of the Compass Health Primary Health Organisation (PHO), which covers the greater Wellington region.

Patients were eligible if aged between 18 and 65 years and identified by their GP in a routine appointment as experiencing stress or distress. Patients were required to score 35 or less on the Kessler Psychological Distress Scale (K10)26 28 during their initial GP consultation, with no lower cut-off on this score. The present study followed previous study protocols24 25 by including scores between 30 and 35 on the K10 as indicative of mild-to-moderate levels of psychological distress rather than major psychiatric disorder. Individuals taking antidepressant or other psychiatric medications were eligible to participate in the study.

Patients were excluded if they lacked fluency in English (as the intervention is an English-language-based ‘talking therapy’); had significant levels of cognitive impairment as determined by the GP or had reported recent or acute suicidal ideation (ie, within the previous 2 weeks). Chronic low level suicidality did not exclude an individual from participating. However, GPs were informed of patients who had high scores or suicidality at screening, or for whom referral to appropriate (secondary) mental health services by GPs was indicated, and these patients were not eligible to participate further in the study.

Inclusion criteria were based on the access criteria of a local partner PHO to psychological therapies. These criteria were youth (aged 18–24 years), or individuals aged 25 years or older with low income, or Māori or Pacific Island heritage.

Recruitment of practices and GPs

Initial recruitment of practices was supported by the partner PHO. GPs were identified using primary health organisation and practice lists. All of the practices contracted under the partner PHO were contacted (n=52) and invited to participate in the study, and an effort was made to contact all of the GPs within these practices by email, telephone or in person. A total of 23 practices initially consented to participate in the study and a further 18 were recruited during the course of the study. Two practices merged and three withdrew (in each case the single participating GP left the practice) leaving a total of 37 practices involved in the study.

Randomisation of practices to study arms

Consenting practices were randomised to provide either UBI or PAU to eligible patients. Randomisation was conducted at the practice level to reduce the risk of contamination if GPs from the same practice were assigned to opposite study arms. To ensure approximately equal numbers of GPs per study arm, randomisation of practices was conducted within five strata, according to the number of participating GPs (one/two/three/four/more than four). An additional two practices dedicated to youth health that were not part of the partner PHO were included and randomised into each arm of the study (ie, these two practices formed their own stratum). Practices were entered into the trial following consent from individual participating GPs in that practice. Randomisation of all consenting practices was conducted following this step by the project biostatistician (JS) using a computer-based randomisation following the above stratification profile.

GPs randomised to the UBI study arm completed a single 2-hour training session (as previously described25). Due to the training nature of the intervention, it was not possible to blind GPs as to their study arm allocation.

Recruitment procedures

GPs identified patients with common mental health problems who might fulfil study criteria during routine appointments. These patients were screened by the GP for eligibility (using the K10), and referred to the study team. A research assistant then contacted potential participating patients, met with them in person where possible to explain the study, confirm eligibility, obtain consent to participate and collect pretreatment (baseline) data. Measures were then collected by mail or email at post-treatment (8 weeks, 3 months and 6 months). Patients received compensation ($NZ30 [US$21] vouchers, and entry into a draw for an iPad) following the completion of the final questionnaire, to recompense for time and effort in participating in the study.

Intervention

UBI is a low-intensity self-management programme which can be delivered by a GP after a single 2-hour training session using a treatment manual based on structured problem solving, motivational interviewing and cognitive behavioural therapy (supported with self-help booklets on relationships, bodily stress, breaking habits and stress management).

Patients who consented and completed the intake data collection (K10 and baseline measurements) received the GP-led intervention in three short, structured face-to-face sessions (one 30 and two 15 min sessions) over a 5-week to 6-week period. Relevant booklets were provided to the patient after the first session, to be used in the following session. In NZ, a stepped care approach to management guides the practitioner towards using the most appropriate therapy option for the severity of presentation. UBI was designed for mild-to-moderate presentations and in training GPs were comfortable with the use of the UBI approach for first-line management. The study protocol allowed for patients in either study arm to alter their treatment as needed (eg, access other talking therapies, or commence mental health medications). Patients were blinded as to their study allocation in that patients in PAU practices were not informed that the UBI was offered in practices randomised to deliver UBI. They were simply told that the study was looking at the effectiveness of PAU.27

Practice as usual

Patients in the PAU study arm received GP support delivered according to their PAU (and available existing services). PAU typically consists of supportive counselling in a 15 min face-to-face consultation, the provision of psychotropic medication, referral to psychological or other counselling options or referral to relevant community services.

Patient characteristics

Patients are described on the basis of age, gender, prioritised ethnicity and NZiDep, a NZ-developed index29 of individual-level socioeconomic deprivation.

GPs in practices assigned to the PAU study arm received optional training in the intervention at the end of the study.

Patient and public involvement

This study had input from an academic mental health consumer (ie, an academic who is also a mental health service user and who conducts research from a service user perspective) as part of the research team at the feasibility stage, and designed the intervention based on feedback from a focus group process with potential patient users of the mental health intervention, which asked what characteristics such an intervention would need to have. This collaborative process is fully described in the study by Mathieson et al.19 This randomised controlled trial did not have academic consumer or patient involvement in the recruitment to and conduct of the study and the burden of the intervention was not assessed by the patients. Results of this study will be disseminated by email to GP participants who indicated they wanted them on the consent form.

Outcome measures

The primary outcome measure was the K10 scale26 28 score at 6 months (adjusted for score at baseline: see the ’Data analysis' section). The K10 is widely used as a clinical outcome measure in primary care and general practice in NZ.9 A 6-month follow-up period was chosen to obtain a sufficient period of assessment following the end of the intervention while at the same time balancing out challenges in patient cohort retention. All analyses were conducted to look at patient-level outcomes.

Secondary outcomes were:

Hospital Anxiety and Depression Scale (HADS), which measures the severity of depressive and anxiety symptoms in outpatient hospital settings.30 Reductions in HADS score indicate reduced anxiety and depression.

Comparison of K10 scores by treatment group at 8 and 12 weeks, adjusted for baseline scores (to capture short-term and medium-term effectiveness).

Work and Social Adjustment Scale (WSAS),31 a measure of work, social and relationship functioning) administered at baseline, 8, 12 and 26 weeks.

Outcomes were measured at the same time points in both UBI and PAU groups (baseline, and at 8, 12 and 26 weeks following baseline).

Statistical methods

Sample size and power analysis

Sample size for the cluster randomised trial was calculated using a simulation method, using SD of patient outcomes from the UBI feasibility study (SD of post-treatment scores=7.525). To detect a difference in K10 improvement scores of 6 points in the UBI arm compared with 2 points in the control arm (at 80% power and alpha=0.05) would require 15 GPs per arm recruiting eight completing patients each on average (n=240 total with complete data). Adjusting for loss to follow-up of 20% gave a recruitment target of 10 patients per GP. The simulation settings roughly correspond to an intraclass correlation (ICC) of 0.15 for considering clustering of patient scores by GP (equivalent to the ICC from the feasibility study25). Power analysis for the secondary HADS outcome indicated 80% power to detect a difference of 3.2 points between groups (based on a SD of approximately 632) assuming a similar ICC for the HADS scale as for the K10 measure (empirical data were not available).

Data analysis

The statistician was blinded to the intervention or control status of participants (both practices and patients) during conduct of the study and analysis. Results were unblinded once analysis was complete. Data processing and analysis were conducted in R V.3.2.3 (R Institute, Vienna) with linear mixed models fit using the lmer package33 and imputation conducted using the mice package.34

For the primary outcome, K10 scores at 6 months were compared between the intervention and control groups using mixed linear models (comparing postintervention scores between groups, adjusting for intake score as a covariate and treating GP clusters as random slope effects). Analysis was conducted on an intention-to-treat basis according to the study arm for each patient at entry into the study. Analyses were adjusted for all other baseline covariates (age, gender, ethnicity, educational level and NZiDep). The original protocol stated that analyses would only be adjusted for baseline values of each score: given some slight imbalance in sociodemographic characteristics it was decided to adjust for other baseline covariates in the main analyses. The originally planned analyses are presented in online supplementary materials 1 (overall patterns discussed in the ’Results' section).

Missing data were handled through the mixed linear models approach to the data, which allows for patients with missing data on the final outcome to be included in analyses, which in effect estimates a final outcome value conditional on the observed data at other follow-up times (ie, validity being predicated under the assumption that the missing observations are missing at random [MAR], conditional on the observed data35 36). Participants missing all follow-up data were excluded from this main analysis. The null hypothesis for this test was that the K10 scores at 26 weeks (adjusted for baseline score) were not different for the intervention and control groups.

Sensitivity analysis for missing follow-up data in the K10 primary outcome were planned and conducted following completion of the main analysis, and hence were not noted in the trial registration or protocol paper. These analyses covered two scenarios: first, an analysis with multiple imputation of missing outcomes, conditional on observed baseline sociodemographics and baseline outcome data. This analysis hence included participants who only had baseline data recorded (excluded from the main mixed models analysis), and assumes that the unobserved outcome data are missing at random conditional on observed data: that is, that individuals who were missing from all follow-up data collections had the same outcome profile (on average) as participants with similar profiles at baseline.37 The second sensitivity analysis explored this missing at random assumption: those missing data postbaseline were (i) assumed to have scores at 6 months that were four points worse than their imputed score in the first sensitivity analysis; (ii) assumed to have had no improvement from baseline (last observation carried forward) and (iii) assumed to have had poorer outcomes at 6 months than at baseline (four points worse than baseline). Full details of the imputation procedure and sensitivity analyses are presented in the online supplementary materials, and results are summarised and discussed in the ’Results' and ’Discussion' sections.

For the secondary analysis, differences in mean scores on the K10 outcome were reported at 8 weeks and 3 months (using the same methods as above, within the mixed linear models framework). Analysis of the HADS and WSAS scores at 8 weeks, 3 months and 6 months used the same methods as for the K10 outcome. Analysis of outcomes at 8 weeks and 3 months was not specified on the clinical trials registry, but was noted in the previously published protocol paper.27

The Euroqol 5D-3L was noted as a secondary outcome for quality of life in the trial registry. This measure was intended as part of an economic analysis that was not implemented, and no other economic data were collected as part of this study.

ICC values were calculated for each outcome measure as a summary of clustering according to GPs. Because our analytical models only accounted for clustering at the level of individual GPs, we also examined ICC values when clustering was considered as a multilevel structure (GPs nested within specific practices). Details of the calculation methods are provided in the online supplementary materials.

Additional treatments received during the trial (including medication and talking therapies) were analysed by study arm, based on self-report data collected at the 6-month follow-up. This descriptive analysis was not specified in the study protocol.

Confidentiality and data management

Consenting patients had their rights explained along with provision for data confidentiality. Paper and digital copies of the data were secured in locked storage on the premises of the University of Otago, Wellington. The questionnaire data were de-identified and entered into a spreadsheet for subsequent analysis.

Ethics approval

Adverse events were not anticipated in this trial, and arrangements were made to feedback clinical information to GPs if deemed necessary (eg, high K10 scores or concerning self-reported statements about a patient’s safety) in the course of data collection.

Results

GP participants

A total of 41 practices agreed to participate, with a total of 112 individual GPs consenting to take part in the study (n=62 for UBI and n=50 for PAU). Of these GPs, 31 recruited at least one patient into the study in the UBI arm (from 22 practices), and 21 recruited at least one patient in the PAU arm (from 12 practices). The numbers of GPs recruiting different numbers of patients is shown in online supplementary table R1.

Patient participants

Figure 1 summarises the flow of patients into the study and participation in the interventions and follow-up. A total of 198 patients were referred into the study, and 160 met eligibility criteria and completed baseline assessments. The vast majority of these completed at least one postintervention follow-up (70/85 in the UBI arm [82%] and 69/75 in the PAU group [92%]) and hence contributed to the data analysis. These patients represented 29 GPs (from 21 practices) and 20 GPs (from 12 practices) in the intervention and control arms, respectively.

Figure 1

Study flow chart of patient participation. GP, general practitioner; PAU, practice as usual; UBI, ultrabrief intervention.

Baseline data

Baseline sociodemographic characteristics of patients are presented in table 1 for the two study arms. The two groups were roughly comparable at baseline, with a few more male participants and a slightly younger age profile in the UBI arm, but with a greater representation of females in the study overall.

Table 1

Patient sociodemographic profile by study arm

Variable	Level	Study group
		UBI (total n=85)	PAU (total n=75)
		n (%)	n (%)
Gender
	Female*	56 (65.9)	57 (76.0)
	Male	29 (34.1)	18 (24.0)
Age group (years)
	15–24	55 (64.7)	37 (49.3)
	25–34	16 (18.8)	15 (20.0)
	35–44	3 (3.5)	13 (17.3)
	45–54	5 (5.9)	6 (8.0)
	55+	6 (7.1)	4 (5.3)
Ethnicity
	New Zealand European /Other	61 (71.8)	54 (72.0)
	Māori	19 (22.4)	14 (18.7)
	Pacific	4 (4.7)	2 (2.7)
	Asian	1 (1.2)	5 (6.7)
Highest education
	At least secondary	78 (91.8)	71 (94.7)
	No secondary level	7 (8.2)	4 (5.3)
NZiDep
	0 (least deprived)	18 (21.2)	11 (14.7)
	1	16 (18.8)	17 (22.7)
	2	15 (17.6)	11 (14.7)
	3	10 (11.8)	10 (13.3)
	4	9 (10.6)	12 (16.0)
	5 (most deprived)	17 (20.0)	14 (18.7)

*Includes one individual self-identifying as female (transgender).

PAU, practice as usual; UBI, ultrabrief intervention.

Mean baseline scores on the outcome measures were also similar between the two groups (table 2, showing means and SD). Box plots of the distribution of baseline scores on each outcome scale are given in online supplementary table R1.

Table 2

Mean (SD) of baseline scores for outcome measures by study arm

Outcome variable	Study group
	UBI (total n=85)	PAU (total n=75)
	Mean (SD)	Mean (SD)
K10*	29.5 (6.2)	28.1 (5.7)
HADS—total	20.6 (5.9)	19.5 (5.1)
HADS—anxiety	12.1 (3.6)	11.9 (3.5)
HADS—depression	8.5 (3.5)	7.7 (3.6)
WSAS	23.0 (8.2)	19.6 (8.5)
Health Thermometer†	55.4 (19.9)	58.8 (18.7)

*One patient in PAU group missing baseline value.

†Higher scores on the Health Thermometer indicate better health.

HADS, Hospital Anxiety and Depression Scale; K10, Kessler-10; PAU, practice as usual; UBI, ultrabrief intervention; WSAS, Work and Social Adjustment Scale.

Health outcomes at follow-up

For the K10 primary outcome at 6 months the mean difference for UBI compared with PAU arm favoured the PAU arm (mean adjusted difference=1.68, 95% CI −1.18 to 4.55; p=0.255: adjusted for age, gender, ethnicity, educational level and NZiDep), as shown in table 3 (where positive differences indicate a better outcome for the PAU than UBI arm). While this result indicated no significant difference in K10 scores between the UBI and PAU arms (figure 2), each group had a reasonable improvement in K10 score from baseline (see supplementary table R2: for the PAU group mean improvement=7.6, 95% CI 5.5 to 9.6; and for the UBI group mean improvement=5.9, 95% CI 4.0 to 7.8).

Table 3

Mean difference in primary and secondary outcomes (difference in change relative to baseline)

Outcome variable	Mean adjusted difference (UBI minus PAU)*
	8 weeks		3 months		6 months
	Mean difference (95% CI)	P value	Mean difference (95% CI)	P value	Mean difference (95% CI)	P value
Primary outcomes†
K10	−0.19 (−2.55 to 2.16)	0.872	1.53 (−0.79 to 3.84)	0.203	1.68 (−1.18 to 4.55)	0.255
HADS	0.57 (−1.68 to 2.82)	0.620	0.86 (−1.38 to 3.10)	0.456	1.85 (−0.62 to 4.31)	0.149
Secondary outcomes†
HADS-A	0.27 (−1.02 to 1.56)	0.684	0.70 (−0.60 to 2.00)	0.296	1.05 (−0.39 to 2.50)	0.161
HADS-D	0.39 (−0.82 to 1.60)	0.533	0.24 (−0.96 to 1.44)	0.701	0.88 (−0.38 to 2.14)	0.178
WSAS	0.49 (−2.40 to 3.38)	0.740	1.32 (−1.58 to 4.22)	0.377	0.45 (−2.47 to 3.37)	0.762
Health Thermometer	2.84 (−3.64 to 9.31)	0.395	1.90 (−4.59 to 8.39)	0.569	4.93 (−1.77 to 11.62)	0.156

*Positive differences indicate better improvement in PAU than UBI arm, adjusted for baseline value of score and age, gender, ethnicity, educational level and NZiDep.

†Number of participants contributing data to each analysis: UBI n=70, PAU n=69 (except for K10: PAU n=68).

HADS, Hospital Anxiety and Depression Scale; K10, Kessler-10; PAU, practice as usual; UBI, ultrabrief intervention; WSAS, Work and Social Adjustment Scale.

Figure 2

Mean K10 score (95% CI) at baseline and follow-up for UBI and PAU study arms. K10, Kessler-10; PAU, practice as usual; UBI, ultrabrief intervention.

The mean adjusted difference on the HADS measure at 6 months between UBI and PAU measures was 1.85 (95% CI −0.62 to 4.31, p=0.149; see table 3), although both groups again showed an improvement in mean score from baseline (supplementary table R1). Mean scores at each follow-up time are presented in figure 3.

Figure 3

Mean total HADS score (95% CI) at baseline and follow-up for UBI and PAU study arms. HADS, Hospital Anxiety and Depression Scale; PAU, practice as usual; UBI, ultrabrief intervention.

Similarly, for all secondary outcome measures (HADS subscales, WSAS and Health Thermometer), the adjusted difference in outcomes at 6 months showed no significant advantage for either UBI or PAU measures (with relatively broad CIs for these differences, see table 3).

Estimates of secondary analyses of outcomes at earlier follow-up times (8 weeks and 3 months) are also presented in table 3. Differences between UBI and PAU were generally most pronounced at the final follow-up (6 months) compared with the interim follow-ups. Trajectories for mean scores in each group are presented in online supplementary tables R2, R3, R4 and R5.

Ancillary analyses

Online supplementary table R3 presents information on types of additional treatment received for those who completed the 6-month follow-up assessment (summary not specified in protocol). Similar proportions of completing patients between study arms were either on medication for mental health condition(s) at the beginning of the trial (UBI=31%; PAU 25%), or started medication during the trial (UBI=18%; PAU=25%). Access to extended GP consultations or counselling sessions was higher for the PAU arm than for UBI (no UBI patient had an extended GP consultation, compared with 29% of PAU patients; and 25% of UBI patients had one or more counselling sessions, compared with 64% of PAU patients).

ICCs for the outcome measures are presented in online supplementary table R4. For the K10 (ICC=0.129, 95% CI 0.045 to 0.231), this was relatively close to the ICC values used in planning the sample size for the study. We also examined clustering effects for GPs as nested within GP practice clusters: this additional complexity (not implemented in our main analytical models) had little impact on ICCs for the K10 or HADS measures, although it did suggest slightly higher ICCs (greater clustering of outcomes than considering GPs alone) for the WSAS and Health Thermometer.

We also conducted three sensitivity analyses for our primary outcome of K10 scores at 6 months. These analyses are described in more detail in online supplementary methods and results.

The first sensitivity analysis used the same linear mixed models analysis as the main reported analysis, but adjusted only for baseline values of the outcome score (as specified in the original protocol: no adjustment for other baseline covariates). This returned a slightly smaller mean difference between study arms (again with a poorer mean K10 score in UBI compared with PAU: difference=1.07, 95% CI −1.67 to 3.82; p=0.447), but does not control for the covariate imbalance seen in recruited participants (as shown in table 1).

The second and third sensitivity analyses both aimed to consider the impact of loss-to-follow-up on the primary outcome analysis, assuming data were MAR or missing not at random (MNAR). Full details of implementation are mentioned in online supplementary methods. Both analyses include all randomised participants. An initial table gives the baseline covariates for those with and without follow-up in the PAU and UBI groups (online supplementary table R5).

The analysis of outcomes under a MAR assumption (including all randomised participants) was almost identical to the main results (online supplementary table R6). Analyses of outcomes under MNAR assumptions were also not substantively different from the main results (online supplementary table R7): the most conservative result returned a mean difference of 2.03 points on the K10 (95% CI −0.63 to 4.70: scenario 1 in online supplementary table R7), which was slightly bigger than the mean difference seen in the main results (1.68 points, as per table 3).

Discussion

The brief psychological treatment (UBI) delivered by GPs in NZ in routine practice settings did not lead to better outcomes than PAU in this pragmatic efficacy trial, with the point estimate for the primary outcome favouring PAU over UBI.

UBI appeared to be slightly less effective than PAU in reducing distress as measured by the K10 (although the difference was not statistically significant). The K10 was originally introduced as an assessment measure of psychological distress, but has also been used to track change in mental health status following intervention.38 There were no significant differences in the secondary measures either.

We were unable to achieve full recruitment to match the predetermined sample size: the study recruited 160 eligible participants across both study arms, against our target of 240 participants with complete data. As such, we were unable to rule out non-inferiority of the intervention (UBI) compared with PAU in reducing the disability and distress associated with mild-to-moderate mental health problems: the bounds of the CIs for the two main outcomes (K10 and HADS measures) included sizable-magnitude better outcomes for PAU over UBI (eg, the upper bound for the K10 was a 4.55 point advantage for PAU).

Both UBI and PAU arms showed improvement in clinical outcome over the 6-month course of the study. These findings are in keeping with other work which demonstrates clinical effectiveness of brief psychological interventions in primary care settings.39

These results suggest that GPs in both arms were achieving clinical benefit. We cannot rule out that UBI performs slightly worse than PAU, but our results are inconclusive due to our reduced sample size. For the last 10–20 years in many OECD jurisdictions, there has been a focus on improving mental healthcare provision in primary care settings. In NZ, this has taken the form of the introduction of locally based primary mental health initiatives, which have increased access to psychological services and provided opportunity for increased engagement (and remuneration) by GPs to undertake mental health consultation work.9 These opportunities were available to the PAU, and may partially explain the relative success of this ‘control’ arm in the study.

Strengths of this study

We consider the results of this trial a useful addition to the literature for two reasons. First they describe the introduction of potentially useful adjuncts to existing therapy approaches in primary care in a randomised controlled setting, and second the ‘negative results’ raise questions about the challenges of conducting pragmatic trials of psychological interventions in primary care and also about the nature and effectiveness of PAU treatments. Feedback received from GPs during the training sessions suggested that elements of the UBI such as active listening, goal-setting; making a specific plan and following up on it are already used in routine practice. UBI had previously been piloted and shown to be both feasible and acceptable to both clinicians and patients in a general practice setting.25 It was also able to be adapted in a culturally responsive way.24 During the course of the trial and following its completion there has been significant interest expressed by both patients and GPs in obtaining copies of the booklets and using elements of the UBI approach in routine consultations. Verbal feedback suggests that GPs particularly liked the helpful/unhelpful behaviour chart which was used to discuss how problems were maintained, the explicit linking of emotional responses to physical symptoms and the use of commitment and capability rulers (a motivational interviewing strategy).

There is an active debate about the optimal balance of intervention components for the management of common mental health problems, with an increasingly varied range of options available. Patients potentially have access to traditional face-to-face intervention with a therapist, access to materials available on the internet and further access to rapidly developing telemedicine and virtual consultation options.40 41 Our study shows that over the course of the trial, patients and GPs were able to adapt the standard pattern of the GP consultation to a series of three sessions, allowing a more participation from the patient. This ability to ‘disrupt’ the traditional pattern of GP consultations is important in an era where there is recognition in NZ and other OECD countries about the need to respond to the changing context of primary care, particularly in relation to long-term conditions including common mental health problems.42

The choice of 4 points for a minimal clinically important difference on the K10 measure was selected on the basis of past work.9 Subsequent research suggests a minimum clinically important difference of around 7 points (measured in younger people accessing services).43 In retrospect, the selection of a smaller difference to detect for the sample size calculation does not affect the interpretation of results as the current study would have had >80% power to detect this revised larger difference between study groups. The original sample size calculation also indicated that full recruitment would have achieved 80% power to detect a difference of 3.2 points on the HADS scale: this was a slightly bigger difference than the minimal clinically important difference cited in the literature.44

We also examined the impact of analytical decisions on our primary outcome, particularly sensitivity analyses examining the potential impact of participants with no postbaseline data (excluded from the main analysis) on the reported intervention effect. There was more loss-to-follow-up observed in the UBI group than in the PAU group. These sensitivity analyses showed relatively little impact on our estimates under several sets of assumptions (online supplementary methods and results).

Limitations

The difficulties in recruiting a sufficient sample size meant we were unable to establish benefit or rule out substantial inferiority of UBI compared with PAU. While we did not meet our recruitment targets, the CIs for our estimates are appropriately wide (reflecting the achieved sample size) and can be taken as valid plausible bounds for the true intervention effect. The main challenges of recruitment for trials in mental health have been described.45–47 The current study contained specific additional challenges as outlined below.

First, our recruitment was limited by specific entry criteria. We would have preferred to include all adults aged 18–65 years with K10s exceeding 35, but our partner PHO was required to limit access to services to clients within the targeted access criteria. This reduced our ability to recruit our planned sample size.

This meant we did not meet our planned sample size target despite energetic problem-solving over a 3-year recruitment period. It also meant that many GPs were not able to recruit any patients (n=60 of the recruited GPs) or were not using the UBI tool until weeks or even months after training. This casts doubt on how well GPs would have adhered to the approach or recalled the principles, potentially affecting the quality of the intervention delivered.

Second, in this NZ context, the GPs in the PAU group had access to a sophisticated range of therapy options which included providing extended consultations themselves, as well as referring patients to psychological therapies such as counselling or CBT delivered by clinical psychologists.9 This introduces the possibility of postrandomisation bias in the control arm due to differential receipt of these other treatments: however, we did not collect details from patients on receipt of such treatments, and thus could not address this potential bias in our analyses. In addition, during the course of the study there were significant changes to the way in which the external psychological services were delivered in our local PHO, with therapists (mental health professionals) being placed within practices rather than at a central location making it easier for in-house referral. Thus, the results may not generalise to settings where these additional therapies are unavailable in day-to-day practice.

These changes made the task of demonstrating non-inferiority more challenging. UBI is consistent with the contemporary primary care stepped care approach that tailors interventions to symptom severity and response to treatment.48 The intervention tool (UBI) used in this study was developed for subthreshold mental health syndromes, but was, in practice, applied to moderate-to-severe problems, due to demand from GPs who said they needed higher thresholds in order to be able to recruit patients. In the NZ context it appears those needing mental health interventions in primary care have more severe problems than the tool was intended for. The intervention may have performed relatively better than PAU if applied to a mild-to-moderate group, but this would need further research to ascertain. The moderate-to-severe group are likely to require longer, more intensive interventions for it to make a difference.

Given the known efficacy of the PAU intervention in this setting,9 the results also attest to the success of the PAU options rather than a specific failing of the intervention. We might expect that clinicians who participated in this study would be those who were motivated and skilled in supporting patients with mental health problems. This is a speculative point, as we did not collect this kind of data on clinician experience, which is a limitation of the study and needs to be considered when thinking about the generalisability of the current results to other settings. It is unclear in this case the extent to which the GPs in the UBI treatment arm were adhering to the structured approach outlined in the treatment manual. Fidelity and adherence to training for psychological intervention has been subject to commentary in the literature49 50 and it is unclear as to the extent to which UBI GPs were able to adhere to the structured manual.

The analyses presented here examined several arising issues that were not planned for at the start of the study. First, there were imbalances on some demographic variables (gender and age group) between the two study arms. While this is suboptimal, the analysis of primary and secondary outcomes adjusted for these and other sociodemographic factors, which means that these imbalances should be accounted for in the results.

Conclusion

In this study, both the PAU and UBI groups showed improvement in clinical outcome, despite UBI failing to demonstrate superiority or conclusive non-inferiority compared with PAU, although the incomplete recruitment means that the precision of our estimates of treatment effects were wide (95% CIs). This leaves open the question of whether this style of intervention may have potential value in a primary care setting, or whether some elements of this style of intervention are already being applied in practice by some clinicians. Our results did not show conclusive evidence that the UBI added value to usual care with patients with moderate-to-severe symptoms, and we were unable to rule out the possibility that UBI patients may have marginally worse outcomes than the control group. Despite this uncertainty in the outcome, the results provide valuable additional information about the provision of brief psychological interventions in primary care.

An ultrabrief approach such as UBI may add value if restricted to patients with mild mental health problems, as part of a suite of options, with different levels of intensity available to GPs in the primary care setting.

There is a significant need for further research into these issues, given the recognition of mental health problems at a community level6 51 and the challenge of providing access to psychological therapy in an effective and cost-effective way.52 53