Systematic client feedback to brief therapy in basic mental healthcare: study protocol for a four-centre clinical trial.

INTRODUCTION: The Partners for Change Outcome Management System (PCOMS) is a client feedback-system built on two brief visual analogue self-report scales. Prior studies of PCOMS have found effects varying from significant positive to negative. Aims of present study are; to test the predicted beneficial impact of PCOMS, while accounting for methodological flaws in prior studies and to clarify under which circumstances the addition of PCOMS to therapy has a beneficial effect. METHODS AND ANALYSIS: This study focuses on patients applying for brief, time-limited treatments. Four centres will be randomised to either treatment as usual (TAU) or TAU with PCOMS. All participating patients will be assessed four times. The full staff in the experimental condition will be trained in PCOMS. In the second part of this study, all therapists in the PCOMS condition will fill in a questionnaire concerning the influence of regulatory focus, self-efficacy, external or internal feedback orientation and perceived feedback validity of PCOMS. Finally, patients in the PCOMS condition will be asked to give feedback through a structured interview.The primary outcome measure is the Outcome Questionnaire over the period from beginning to end of therapy. The Mental Health Continuum-Short Form and Consumer Quality Index are also completed. In the primary analysis, outcomes of the two treatment conditions on treatment outcome, patient satisfaction, costs, drop-out and duration will be examined with a three-level (within patient, between patients and between therapists) multilevel analysis. The DSM-classification, sex, education level, age of each patient and therapist factors will be included as covariates. ETHICS AND DISSEMINATION: The Medical Ethics Committee of the University of Twente approved this study (K15-11, METC Twente). Data will be included from 1 January 2016 to 1 July 2019. Study results will be disseminated through peer-reviewed journals and conferences. TRIAL REGISTRATION NUMBER: NTR5466; Pre-results. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

Strengths of the study are the natural setting, good implementation-strategy and the large sample sizes.

In this study, therapist factors and clients perspective are also taken into account.

Treatment progress of this feedback intervention is measured with an independent outcome measure.

The relatively small number of participating centres might lead to limited generalisability.

Background

Reductionism and mental disorders

Reductionism is the philosophical view which holds that the nature of complex entities can always be traced back to more basic entities. Although reductionism is a prerequisite to order our world in comprehensible elements, there is always the risk that relevant areas are neglected. In psychotherapy, there are two major reductionist examples that play an important role in daily routine. These are the Diagnostic and Statistical Manual of Mental Disorders (DSM), nowadays in its fifth edition1 and the Practice Guidelines on mental healthcare. They both can be helpful. The DSM provides therapists with ways to look at (psychiatric) disorders and the Practice Guidelines guide them towards a variety of potentially helpful interventions.

There are two main downsides to these two reductionist concepts: (1) its focus on disorders instead of on well-being and (2) its focus on groups of patients, while neglecting specific patient characteristics and specific patient–therapist interactions at the same time.

The problem of focusing on disorders instead of on well-being

Mental disorders, the main focus of the DSM, always have an inseparable personal, relational and social context and are not autonomous entities. Therefore, most people only seek therapy when the disorder hinders them in their well-being on a personal, relational or social level.

According to Frank,2 people apply for therapy when they become demoralised, and their own solution strategies are not sufficient. Therapy has the purpose to enable people to improve functioning and to facilitate growth. Many psychiatric disorders can only be cured to a certain degree or cannot be cured at all, but an improvement in well-being (with or without curing the disorder) can, on the other hand, be achieved most of the time. Therapy and therapy outcome instruments should therefore not (only) focus on psychiatric complaints but also on well-being. In the DSM 5, (impairment of) functioning is categorised in the WHO Disability Assessment Schedule,3 but the emphasis remains on psychiatric diseases.

The problem of focusing on groups of patients, while neglecting specific patient characteristics and specific patient–therapist interactions at the same time

The practice guidelines on mental healthcare provide us with relevant information about groups of patients (for instance, severely depressed patients), but do not give specific information about the individual patient. To create tailor-made treatment for an individual patient, it is necessary to modify therapy to specific individual needs. There are some protocols for more tailor-made treatment,4 but the majority of treatment programmes are still organised based on what works for groups of patients with a specific disease.

Apart from creating tailored treatment, the effectiveness of therapy also depends on the quality of the working alliance, having a robust effect of approximately 7% on treatment outcome throughout different kinds of therapies.5 Bordin6 distinguished three main features of the working alliance: agreement on goals, assignment of tasks and the development of bonds.

PCOMS, combining randomised control trials with a clinical reasoning tool

The famous Lambert group of the Brigham Young University7–9 conducted a series of large randomised controlled studies on the added value of providing systematic patient feedback on treatment progress, using the 45 item Outcome Questionnaire (OQ-45). These pioneers found that presenting systematic progress feedback to therapists on patients’ actual OQ-45 scores (compared with their previous OQ-45 scores, and compared with other patients’ OQ-45 scores) substantially reduces the percentages of deteriorated patients, and increases the chances of return to normal functioning and improve well-being.

PCOMS is intended to be a lean and simplified variant of the original OQ-45. Duncan and Sparks10 explored how, with the least possible effort, systematic feedback could increase the effectiveness of therapy. Miller, Duncan, Brown, Sparks and Claud11 proposed the brief (only four items) Outcome Rating Scale (ORS), adapted from the OQ-45. Rather than assessing symptoms or complaints, the ORS taps the domain of well-being in daily life, distinguishing between the intrapersonal, relational (interpersonal) and societal level (social roles [SR]). The ORS purports to broaden the focus of therapy and monitors progress on well-being which is in accordance with the primary goal of therapy: optimising health. The same group developed the Session Rating Scale (SRS)12), a brief scale covering the three alliance features as defined by Bordin: (1) the bond, (2) goals and topics and (3) approach or method. Using the SRS for feedback could optimise the working alliance.

This method of systematically using the ORS and SRS for feedback during each therapy session, is referred to as the Partners for Change Outcome Management System (PCOMS). Adding PCOMS to treatment as usual (TAU) combines the expertise and knowledge obtained by several randomised controlled trials (RCTs) on the one hand with a clinical reasoning tool (CRT) on the other. In this way, therapy is optimised for a specific client by focusing on improving of well-being and on the working alliance.

There is evidence that PCOMS can be a useful addition to TAU, in accordance with the Practice Guidelines on mental healthcare (see review article by Duncan and Reese)13. It can augment treatment effectiveness, as well as efficacy, and can decrease dropout rates. Duncan and Reese found five RCTs that support the efficacy of PCOMS with overall effect sizes ranging from d=0.28 to 0.54.

However, these studies had some methodological flaws. In three of those RCTs, no independent outcome measure was used14–16 leading to contamination of ‘predictor’ and ‘outcome’. In addition, the training of the therapists varied considerably, ranging from no training at all in the study of Schuman et al 17 to an 8-hour training followed by three 3-hour follow-up trainings.18 One RCT in which an independent outcome measure was used, found no significant effect on that measure.17 Anker, Duncan and Sparks reported a significant effect in their RCT study on a non-validated internal measure administered by therapists treating military clients, or their commanders. However, no effect was found for clients’ self-ratings.18

Not all studies had these contamination flaws; a cohort study19 and a benchmark study20 using an independent outcome measure, found significant positive effects. Findings of three other RCT studies with an independent outcome measure vary considerably in effect. Janse et al found no beneficial effect for PCOMS in treatment outcome but reported a positive effect on efficiency (12.5% cost reduction) of cognitive behavioural therapy (CBT).21 Rise et al found no short and long term effect of PCOMS on treatment alliance and patient satisfaction but they found a positive effect on treatment motivation.22

A negative effect of PCOMS was found by van Oenen et al.23 24 These authors were studying patients with severe distress at a Psychiatric Emergency Centre. Their patients initially scored 13.3 on average, while the ORS clinical cut-off score is 25. They concluded that patients with lower ORS-scores at the start (other studies reported initial scores varying between 18.3 and 23.7) may benefit less from adding PCOMS to therapy. However, there were some adherence weaknesses: the principles of PCOMS in this study were explained to all patients by a research assistant instead of the therapists and the ORS was scored before each session and not in presence of the therapists.

The circumstances under which PCOMS can have a beneficial effect and its working components still remain unclear. In a dismantling study, Mikeal et al 25 found that using either the ORS or SRS component of the PCOMS, or the full PCOMS produced equivalent outcomes, suggesting that PCOMS is not particularly a measurement tool, but rather a CRT improving the communication between therapist and patients. However, this study had no control group with TAU to clarify if PCOMS had any effect at all. Anker et al 18 reported that the effect of feedback varied significantly across therapists; less effective therapists benefitted more from feedback.

In a meta-analysis of 12 studies (only partially involving PCOMS) including 3696 participants which used patient-reported outcome measures, Kendrick et al 26 found no evidence of a difference in outcome in terms of symptoms between feedback and no-feedback groups. They concluded in their Cochrane review that further study is needed and systematic feedback does not always or automatically lead to improvement. In conclusion, although some studies confirm that PCOMS improves the effectiveness and/or the efficacy of TAU, the circumstances under which PCOMS can have an additional value still remain unclear.

In the Netherlands, PCOMS has been studied on a small scale.21 23 24 27 Hafkenscheid provided the first translation of the ORS and SRS into Dutch28 and called PCOMS Routine Process Monitoring. His version was psychometrically validated by Hafkenscheid, Duncan and Miller.29 A second translation of both instruments into Dutch has been made available by Asmus, Crouzen and Van Oenen. In the present study, the second translation will be used, since the Asmus et al translation has been claimed by the legal owners as the only officially approved version. Its psychometric properties were studied by Janse et al.30 The psychometric results of both Dutch versions turned out to be approximately comparable.

Aim of present study

The aim of the present study is to clarify whether and under which circumstances the addition of PCOMS has a positive effect on the treatment outcome for therapy offered in basic mental healthcare in the Netherlands. The impact of PCOMS on treatment outcome will be measured in terms of effectiveness of treatment, patient satisfaction, drop-out and cost-effectiveness. The primary outcome measure is the OQ-45.

To augment the understanding under which circumstances the addition of PCOMS has a positive effect on treatment outcomes, therapist variables will be measured in terms of self-efficacy, regulatory focus, external or internal feedback orientation and perceived feedback validity. Some patients selected in the feedback condition will be asked to give feedback through a structured interview.

The study was waived the need for informed consent stating that PCOMS is a minor procedural intervention and that patients are therefore not subjected to proceedings or that a certain behaviour is imposed on them. In addition, the METC approved that data collection of the outcome measurement on this study could be obtained if patients do not object against using Routine Outcome Measurement (ROM) data for scientific research.

Method

The study is a four-centre clinical trial in basic mental healthcare, in which all patients will receive brief therapy (BT). This clinical trial is reported according to the Standard Protocol Items: Recommendations for Interventional Trials guidelines (see online supplementary additional file 1). Adding PCOMS to BT cannot be tested ‘double blind’. When therapists are trained in applying PCOMS, it might well influence their therapeutic behaviour in general, even in sessions when they do not add PCOMS. Randomisation is therefore secured at the level of the participating centres and not at the level of the patients. In this way, all centres and all therapists either never use PCOMS or always use PCOMS, lowering the chance of control group contamination.

Four centres have been randomised to the experimental condition (BT-PCOMS) or the control condition (BT), as presented in figure 1. The full staff within the experimental condition was trained in PCOMS in an interactive programme which takes one and a half days.

Figure 1

Study design.

Over a period of 2 years, all included patients will be followed, starting in January 2016. The patients enrolled in the experimental condition will receive TAU combined with PCOMS. The patients treated in the control condition will be offered TAU. The inclusion criteria for patients are: 18 years or older, and assigned to and receiving psychological treatment in basic mental healthcare. They must master Dutch as their first or second language and do not object against using data obtained by ROM for scientific research. There are no exclusion criteria other than failure to meet one or more inclusion criteria.

Basic mental healthcare

In the Netherlands, treatment of patients with mild to moderate, non-complex psychological disorders with a DSM classification takes place in basic mental healthcare. Patients with complex disorders are referred to specialised mental healthcare. In basic mental healthcare, BT is offered, which is defined by a maximum of 12 sessions. All patients of this study are treated in Mindfit, one of the largest organisations of basic mental healthcare in the Netherlands. Mindfit has more than 200 therapists working in 17 self-managing teams spread over more than 40 locations, mainly in the eastern part and in the centre of the country. Their therapeutic approach is based on (positive) cognitive behavioural therapy31 and solution focused therapy.32

Recruitment

Recruitment of the therapy centres took place during a meeting of Mindfit. All centres were invited to participate in the study. Four centres agreed to participate. These centres are based in Kampen, Steenwijk, Zutphen and Zwolle(-West) and all centres have between eight and ten therapists (figure 2).

Figure 2

Participating centres.

Kampen, Steenwijk and Zutphen are all small towns, with about 60 000 inhabitants each, including the rural community. Zwolle, capital of the province of Overijssel, is a middle sized city, with approximately 125 000 inhabitants, served by two centres of Mindfit: Zwolle-West and Zwolle-Oost (East). The centre Mindfit Zwolle-West participated in this study. Each of the four centres covers a healthcare area with approximately 60 000 inhabitants. The average annual income per household for all four regions is approximately the same (33 000€, see reference 33).

All four participating centres were measured and compared in the first half of 2015, before the study started, to ensure that their treatment effect prior to this study was comparable. This treatment effect was measured with rate of change on the Korte Klachten Lijst (KKL), a Dutch validated treatment outcome scale.34 This scale was used because that was their only instrument for Routine Outcome Monitoring until September 2015. The KKL contains 14 items, with each item representing one of the following symptoms: fears, concentration problems, memory problems, depression, physical complaints, irritability, relationship problems, suicidal thoughts and/or suicide attempts, eating disorders, automutilation, sexual problems, sleeping problems and addiction. The total score is determined by adding the answers to questions 1–13 together. The 14th item is an open question. The respondent can fill in a complaint which does not occur in the fixed items.

Improvement on the KKL, in the final measurement compared with the baseline, showed no significant differences for all four centres, and also no significant differences with the overall Mindfit organisation in the year prior to the study (see table 1).

Table 1

Treatment effect delta t score of the Korte Klachten Lijst of participating centres in 2015

Centre	N	T-beginning–T-end	Delta t score (min–max)
Steenwijkerland	224	48.30–39.93	8.37 (7.94–8.80)
Kampen	196	47.77–40.58	7.19 (5.87–8.51)
Zwolle West	133	47.18–37.92	9.26 (7.72–10.81)
Zutphen	234	47.87–38.04	9.84 (8.63–11.05)
All centres (n=13)	1868	48.27–39.91	8.36 (7.93–8.79)

In September 2015 the centres started using the OQ-45, MHC-SF as well as the KKL for ROM. In 2016, all centres started using only the OQ-45 and MHC-SF for Routine Outcome Monitoring.

The board of Mindfit actively supported the study. They were open to adding PCOMS to TAU but wanted to study its effect and best implication strategy. The four Mindfit centres willing to participate agreed to deal with the demands of the study. This included production loss due to the training, requirements for data collection, willingness to be randomised, and acceptance of a 50% chance of being trained immediately, a 50% chance of a delay of training (at the end of this study) and a 100% participation of individual professionals in multidisciplinary training sessions. Each centre was already recording the DSM classification of each patient, treatment duration in minutes, sex and age of each patient, reason for therapy ending and number of registered professionals, as well as ROM as part of their daily routine prior to the study. All patient outcome variables are therefore gathered automatically as part of TAU. Patients are informed that the acquired anonymised data can be used for scientific purposes. In addition, they can object to the usage of their data in scientific research and then their data will not be included in the study. Under Dutch law, this is legal, and standard for most mental healthcare centres in the Netherlands.

PCOMS is a brief method. Patients fill in two digital four-item visual analogue scales each session and patients are free to stop using PCOMS at any time. Moreover, in one of the scales the patient is specifically asked what the therapist should maintain, increase or stop in therapy, including the PCOMS method.

Randomisation procedure

The four participating centres were randomised into the two conditions: BT with PCOMS (BT-PCOMS) and BT which was TAU. An independent researcher not involved in the study carried out the randomisation, using the http://www.randomization.com website. Results of the randomisation can be found in table 2.

Table 2

Results of randomisation of the participating centres

Centre	Condition	Number of professionals	Psychiatric nurse	Psychologist
Kampen	BT	8	2	6
Steenwijk	BT-PCOMS	8	1	7
Zutphen	BT	8	2	6
Zwolle-West	BT-PCOMS	10	6	4

ROM is used by default in mental healthcare in the Netherlands and these outcomes are discussed in session as part of TAU. The primary purpose of national ROM is a collection of treatment data for all patients in Dutch mental healthcare, with the OQ-45 as one of the major efficacy measures. One of the ultimate secondary objectives of Dutch national ROM is benchmarking between individual therapists and different mental health agencies. The data in the present study will be collected within a period of time from 1 January 2016 to 1 July 2019.

Sample size calculation

Duncan and Reese found five RCTs that support the efficacy of PCOMS with overall effect sizes ranging from d=0.28 to 0.54.13 De Jong et al however, found smaller overall effects of feedback on patient progress (d=0.19) in a meta-analysis of several feedback methods.35 Therefore the effect of PCOMS in this study has been estimated to be small (Cohen’s d=0.2) in the power analysis.

This study includes only four centres, so the data will contain only scant information about the basic healthcare population. A common approach would be to treat the centres as a random effect. Due to the small number of centres, they are not representative for the population. Moreover, when there are only four centres participating, a realistic estimation of the between-centre variance also cannot be made. It is therefore recommended to include the centres by using fixed rather than random effects in the statistical model (see Section 4.3.1 of reference 36). In other words, we do include the centres by using a centre level in a multilevel model, as is common in cluster randomised trials, but fit a model that uses dummy variables to represent the centres.37 The section that describes the statistical analysis gives further details on how to specify these dummies. As a result of this, the SE will be smaller leading to a higher chance of type one error. Generalisability is also hindered.

To calculate the sample size to achieve sufficient power at posttest we use equation 12.3 from.38 This equation relates power to effect size, type I error rate and SE of the treatment effect estimate. This SE in its turn depends on the sample size and follows from table 1 of Moerbeek et al,39 with the variance component at the centre level fixed to zero because centre is treated as fixed. We distinguish three levels of nesting: clients within therapists, within centres. The intraclass correlation coefficients at the therapist and centre level are defined as the proportion of the total variance that is located at these levels (see equations 2.16 and 2.17 in reference 38) and are estimated to equal to 0.027 and 0.039, respectively.

This intraclass correlation was acquired from data from the same four centres with the same therapists in the period prior to this study, dating from September to December 2015 involving 221 patients. The same primary outcome measure (OQ-45) was used. Our calculations are based on a sample of eight therapists per centre. To achieve 80% power in a test with a two-sided alternative hypothesis, type I error rate α=0.05 and a small effect size (d=0.2) we need 26 clients per therapist, or a total of 208 clients per centre. This is approximately half the number of patients each centre will treat in this period. The four centres have no waiting lists.

Intervention

Within the experimental PCOMS condition, each patient is asked to complete the ORS. The ORS encompasses the domains of actual personal, interpersonal, social and overall levels of well-being as experienced by the client. Each domain is scored on a 10 cm line in visual analogue format. The ORS is administered digitally, all patients are using the therapist’s desktop computer or laptop. Mindfit has an excellent office automation service guaranteeing almost 100% device access and internet connectivity. The patient clicks with a mouse at a specific place on each scale on the screen that represents his or her perception of well-being in that particular area of life.

The ORS is completed at the beginning of each session, and the client’s scores are discussed instantly. The conversation topics of the remaining session are determined on the basis of this discussion, departing from the question ‘What should we discuss today so that your degree of well-being on one or more of these categories will improve?". If a client assigns higher ratings on the ORS score, the therapist enquires what efforts the client has performed in relation to the reasons for service to improve things, making sure to help the client take responsibility for the noted changes. If the ORS is not showing improvement, the therapists is signalled and that information is ‘fed back’ into the session with questions like ‘do we need to do anything different today to address your reasons for service?”. The longer the lack of progress continues, the more the therapist redirects toward a discussion of options and different treatment avenues including switching to a different therapist.

The four-item total score is plotted into a graph. Therefore, the discussion is not only about how things are going with the patient at that particular session, but also the total score is compared directly to the results of all previous sessions. The course of these scores, especially when there is no upward trend, a stagnant growth or a sudden bend in the course, will also be discussed with the patient.

At the end of each session, the client completes the digital SRS. The patient’s responses to the SRS items are discussed immediately. The SRS also consists of four, 10 cm long, visual analogue scales. They represent the elements of the working alliance as formulated by Bordin5: client’s feelings about the bond, the topics that were discussed (goals), and the ways followed to attain goals (tasks). The fourth item concerns the perceived overall quality of the session.

Based on the clients’ ratings, the participants discuss what the therapist should maintain or increase and what the therapist should do better or differently in the next session. To reduce risk of social desirability of clients’ responses, as identified by Hafkenscheid,40 therapists are trained to specifically elicit negative feedback. The more negative feedback can be generated, the more treatment adaptations can be made, and the more therapy can be customised to the needs and desires of the patient. The total scores on the four items of the SRS and ORS are presented in the same graph, enabling a direct comparison of ORS and SRS ratings over the course of treatment. ORS and SRS ratings of the present session can directly be compared with the results of all previous sessions.

If SRS scores stay low and stagnate after three or four consecutive sessions, the other therapists of the team will be consulted. If no score changes occur even after this consultation, the patient will be provided with the opportunity to switch to another therapist.

Moreover, not only the score pattern of the SRS over time is discussed. Bidirectional comparisons are also made between the score patterns of the ORS and SRS over time. If for instance the SRS scores are consistently high, whereas the ORS scores stay low, then the patient expresses satisfaction with the therapy but conveys the message that therapy does not improve well-being. Of course, this poses a problem, since one of the major goals of therapy is to improve well-being in their own lives.

Adherence

Therapists in the feedback condition received an 8 hour training of PCOMS (see content above) followed by 4 weeks of practice and then a 4-hour follow-up training. Training was given by an experienced instructor not participating in this study who has been trained by the founders of PCOMS and is experienced in both the use and training of PCOMS. Training followed the protocol of PCOMS and therapists received the Dutch manual of PCOMS translated by Crouzen.41

After this training, all therapists practiced for two more months in using PCOMS in clinical practice. This enabled them to become familiar with applying the method and digital administering. After this total training period of 3 months, the first patient was included. During the training period and whole study, therapists are given supervision every four to 6 weeks by the lead researcher who is also trained by the founders of PCOMS and who did not participate as a therapist in this study.

In both the experimental and control conditions each team is given supervision every 4– 6 weeks by the lead researcher, to avoid bias. Supervision consists of a 1-hour supervision session with all team members, where therapists are given the opportunity to review their work with patients. In the BT-PCOMS condition therapists are encouraged to also show the score patterns of the ORS and SRS over time, so these can be discussed too. This supervision also enables reporting, and managing solicited and spontaneously reported adverse events of PCOMS. The instructor can be contacted during the whole study.

There are no Dutch norms for PCOMS used in the basic mental healthcare. To ensure that therapists pay extra attention to those patients who show no change or deterioration, red and green arrows were used (see figure 3).

Figure 3

Red arrow down and green arrow up.

A red arrow pointing down is shown at the results every time a patient stagnates or deteriorates on the ORS score compared with the previous session. If a patient makes progress, a green arrow pointing up is used. As stated before, this is a four-centre clinical study and outcome data are obtained as part of TAU. Therefore, a fidelity measurement, such as audio review of for instance 20% of the sessions, could not be obtained because it would disturb its naturalistic setting. In the data-analysis however, the missings will also be taken into account and a sub-group analysis will be conducted on those patients in the PCOMS-TAU condition who used PCOMS in at least 75% of their sessions. In this study, fidelity will be defined as having completed at least three full sessions with PCOMS. A sub-group analysis will therefore be conducted with those patients who had at least three sessions.

Measurements

To assess the treatment outcome, the OQ-45 and the Mental Health Continuum Short Form, Dutch version (MHC-SF) will be used in the present study. These instruments both measure psychiatric symptoms and well-being. Symptom-based measures may result in different outcomes than measure of well-being, particular to the patient’s reasons for service.

The OQ-458 is one of the most commonly used outcome measures in clinical psychology and psychiatry, both in the Netherlands42 and in the USA.43 Treatment progress is measured on three dimensions: Symptomatic Distress, Interpersonal Relations and Social Role. The OQ-45 uses a five-point scale ranging from 0 (never) to 4 (almost always). The total score ranges from 0 to 180, and is calculated by adding all the scores on all 45 items together. The higher the score, the more problems the patient reports.

Reliability and validity of the Dutch translation has been established by de Jong et al,44 45 demonstrating that the psychometric properties of the OQ-45 are satisfactory and comparable to the original American instrument. However, an additional Anxiety and Somatic Distress scale emerged from their analyses of the Dutch data.46 The dysfunctional score range for the OQ-45 is empirically set at a score equal to or exceeding 56. An OQ-45 score difference of 18 points for repeated administration of the scale is considered to be statistically reliable.47 The correlation between the Dutch ORS total scale and the Dutch OQ-45 is −0.62.30

The MHC-SF is a self-report questionnaire measuring treatment outcomes in terms of positive mental health. The MHC-SF is based on a long version (MHC Long Form), which consists of 40 items.48 Its short form contains 14 items and includes three components of well-being: Emotional (3 items), Psychological (6 items) and Social Well-being (5 items). The items are scored on a 6-point scale ranging from never (0) to every day (5). The sum of the three scale score represents the respondent’s experience of overall positive mental health. Emotional well-being is about life satisfaction and positive feelings, such as happiness, interest and pleasure in life.49 Psychological and social well-being focus on the optimal functioning of people rather than on the experience of satisfaction and happiness. Psychological well-being focuses on optimal personal performance and includes aspects such as autonomy, connectedness and self-acceptance.50 Social well-being focuses on optimal functioning in society, such as social contribution and integration.51 The Dutch version of the MHC-SF has been empirically tested by Lamers et al.52

Earlier studies of PCOMS have suggested that adding PCOMS to therapy could lead to higher patient satisfaction, lower therapy costs and reduction of dropout.13 The Consumer Quality Index (CQI) short version for mental healthcare will be used to measure patient satisfaction. The CQI is a science-based, standardised methodology for measuring customer experiences in care. This index is developed by the Dutch Trimbos Institute for research into mental health issues. The psychometric properties of the CQI were studied by Wijngaarden et al.53 Its topics are interpersonal conduct, accessibility to the therapist, received information, (therapy) options and an overall rating. The therapy dropout percentage, costs and therapy duration will also be assessed. The various assessments and questionnaires are summarised in figure 4.

Figure 4

Summary of assessments.

As was mentioned in the background-section of this article, therapy is intended to improve well-being, with or without curing the disorder. Well-being can be improved by the reduction or minimisation of symptoms and disorder, but it can also be improved by the presence of positive emotions, life satisfaction and striving for psychological well-being and social welfare, regardless of the absence or presence of a disease. The reduction of symptoms and improvement of well-being will be measured with the OQ-45. The improvement in well-being will be measured with the MHC-SF. This is in line with the WHO54 who advocates not to limit the definition of mental health as the absence of mental health problems. The more positive elements of health: being able to utilise your talents, having meaningful relationships, and being able to make a social contribution, can be most rewarding and can also help to carry the weight of an actual disease.

A form of feedback is obtained by default in both conditions through administrating the OQ-45 and MHC-SF four times for ROM. In Dutch mental healthcare, ROM is used more as a tool for benchmarking than as a process monitoring instrument. The use of the OQ-45 and MHC-SF is not accompanied with clinical support tools. Therapists are not trained in using ROM to monitor and evaluate treatment nor to flag Not On Track patients. Since ROM is used by default in both conditions, but primarily for benchmark purposes, it aims to be a small confounder. Moreover, the frequency of feedback differs strongly in the BT-PCOMS condition where added feedback is given twice per session on functioning as well as on the working alliance. This feedback is therefore much more frequent, used for process monitoring and more diverse. In the BT-PCOMS condition therapists are given specific criteria on how to handle when there is no progress.

Therapist characteristics study

A study of therapist characteristics will also be conducted in line with the recommendations of Miller et al 55 who stated that the therapist’s contribution plays a vital role in therapy outcome. PCOMS as a contextual, ‘general factors’ method, focuses on the personal, relational and social environment in which diseases or disorders occur. Therefore, we should also focus on the personal, relational and social environment of the therapist using this method. De Jong and De Goede,56 for instance, argued that personal characteristics of a therapist could affect the success of using outcome feedback. They concluded that therapists with a strong tendency to prevent failures (prevention focus) had a more positive attitude toward feedback, but achieved slower symptom reduction in their at-risk cases. Therapists with a strong tendency to achieve success (promotion focus), on the other hand, had a faster symptom reduction in at risk patients when feedback was provided. This phenomenon is known as the regulatory focus theory.57

Self-efficacy, external or internal feedback orientation and perceived feedback validity also influence the effect of feedback.58–60 Therapists with high self-efficacy have more belief that their efforts will improve outcome than therapists with lower self-efficacy who will more likely lower therapy goals when they get negative feedback. Therapists who are more external feedback orientated usually have more faith in external feedback, while more internal feedback oriented therapists disregard external feedback. Finally, the perceived credibility and informational value of the feedback (feedback validity) will affect the receiver.

The aim of this study is to analyse the PCOMS and under which conditions PCOMS is helping or hindering. All participating therapists will fill in a questionnaire consisting the Internal and External Feedback Propensity Scales,61 an adaptation of the CFIT User Survey (about self-efficacy and perceived feedback validity)55 and an adaptation of the general regulatory focus questionnaire.57 Therapists will also be asked to estimate which percentage of sessions they actually applied PCOMS and estimate how much time on average they used to implement the results shown by ORS and SRS in the therapy. The results of this questionnaire will be related to the results of the main study to find out whether regulatory focus, self-efficacy, feedback orientation (external or internal) and perceived feedback validity influence the effect of feedback. In this part of the study, we also include all intention to treat patients to examine whether therapist factors have an impact on properly conducting PCOMS.

Patients perspective

During the study, some patients selected in the feedback condition will be asked if they would be willing to give feedback through a structured interview, based on a study of Boeschen62 about their experience of using PCOMS. This method of investigating is in line with the suggestions of Kramer63 who underlines that not only mediators (why a treatment might be preferred to the control condition) should be investigated but also moderators (under which conditions treatment choice differentially affects outcome).

Statistical analysis

All patients will be assessed four times (see figure 4), at the beginning of the therapy, after 5 weeks of therapy, after 13 weeks of therapy and at the end of therapy. The OQ-45 and MHC-SF will be administered in all four measurements. The CQI will be added in the final measurement. In the primary analysis, outcomes of the two treatment conditions on treatment outcome (OQ-45 and MHC-SF), patient satisfaction (CQI), costs, duration and drop-out rate will be compared using a repeated measures MANOVA. The DSM classification, sex, education level and age of each patient, therapist characteristics and the total duration will be included as covariates. The primary outcome measure is the OQ-45.

This will be examined with a three-level (within patient, between patients and between therapists) multilevel analysis. The CONSORT Statement on Group Randomised Trials64 describes that each condition should have at least four participating centres for a valid cluster randomisation. Unfortunately, we were not able to recruit more than four participating centres in total. Due to this small number, a realistic estimation of the error variance also cannot be obtained. Therefore we use fixed effects regression to correct for the clustering in that way.34 With this approach, there is one reference centre per treatment condition and a fixed effect is included for the other centre within that condition. This dummy has the value +1 for that centre and −1 for the reference centre. This will lead to a smaller SE than a regression that treats centres as random, and generalisability is also hindered. In the secondary analysis, characteristics of the therapist and the team will also be examined with a three-level (within patient, between patients and between therapists) multilevel analysis. In the first part of this study, primary outcome will be the OQ-45. Secondary outcomes are the MHC-SF, patient satisfaction, costs, drop-out, and duration. In ideal test conditions, fidelity of PCOMS could be defined as minimally 80% application of the method. In a natural setting, a dose-effect relationship analysis of using PCOMS is more appropriate. The number of face to face sessions in relation to the number of submitted ORS and SRS will be systematically registered for each patient. In sessions where the OQ-45 and MHC-SF will be discussed as part of TAU, the SRS and ORS will not be completed. A sub-group analysis will be conducted with those patients who had at least three sessions and with those patients in the PCOMS-TAU condition who used PCOMS in at least 75% of their sessions.

Patient and public involvement

Mindfit is part of a larger mental health organisation called the Dimence Group. The Dimence Group has a patient advisory board called ‘cliëntenraad’. In July 2015, this study was presented to a delegation of this cliëntenraad and asked for advice. The PCOMS method was also demonstrated. The full delegation supported the study, stating that this study was a small burden for patients involved and could help to improve shared decision making. They had no suggestions for improvement in the design, recruitment or conduct of this study or other suggestions for improvement. The results of this study will be presented in the patient magazine of the Dimence Group and to the cliëntenraad. Patients who are interviewed are offered to receive a copy of the manuscript describing this part of the study.

Discussion

This clinical study aims to measure the effect of implementing PCOMS in a naturalistic setting. The effect on treatment outcome will be measured in terms of patient satisfaction, drop-out, effectiveness of treatment and cost-effectiveness. The primary outcome measure is the OQ-45. Strengths of the study are the implementation strategy and implementation feedback, the training of the full staff and the willingness of the staff to participate in the study, adequate statistical power and the use of an independent outcome measure which is not used in the intervention. Therapist factors and clients perspective are also taken into account. It is expected that the study will give more insight into the applicability of PCOMS in basic mental healthcare and BT.

Weaknesses of the study are the predictably relatively large number of missing data, the small number of participating centres leading to hindered generalisability, as well as relatively long period of data collection. The fact that PCOMS will not be used in sessions where the OQ-45 and MHC-SF will be discussed, can also be considered as a weakness. However, asking for feedback twice (by discussing the OQ-45 and MHC-SF followed by PCOMS) is too burdensome for the patient and could hinder the working alliance.

When PCOMS appears to be a useful intervention in terms of increased or accelerated symptom reduction and/or improvement of well-being, fewer treatment sessions would be needed to obtain the same treatment outcome. PCOMS could also reduce the number of drop-outs and/or increased patient satisfaction, all leading to cost reduction. PCOMS therefore could be highly cost-effective, considering the low cost of training.

Trial Status: Data will be included from 1 January 2016 to 1 July 2019.