Assessment of shared decision-making in community mental health care: Validation of the CollaboRATE.

BACKGROUND/
OBJECTIVE: CollaboRATE is a 3-item self-report measure of the patient experience of shared decision-making (SDM) process. The objective of this study is to assess the psychometric properties of CollaboRATE in community mental health care.
METHOD: A cross-sectional study was conducted at a Community Mental Health Center of the Canary Islands Health Service. Two hundred and fifty consecutive psychiatric outpatients were invited to participate. Of those, 191 accepted (76.40% of response rate) and completed the CollaboRATE, the Control Preferences Scale (CPS), and a form with sociodemographic and clinical variables.
RESULTS: Exploratory factor analysis ratified the unidimensionality of the measure. High internal consistency was found (α Cronbach = .95, Guttman's λ = .93, and ω = .95). Strong positive correlations (p < .0001) were found between the CollaboRATE and the CPS. Only 39.80% of respondents gave the best possible score on CollaboRATE.
CONCLUSIONS: This study provides evidence for the reliability and validity of the Spanish version of the CollaboRATE as a measure of SDM. The measure is quick to complete and feasible for use in outpatient mental health care. At present, a significative number of psychiatric outpatients are not involved in SDM. The use of this measure in psychiatric routine care can be a key tool in assessing and implementing SDM.

Shared decision-making (SDM) is a collaborative and mutual process in which patients and clinicians address the patients’ values and preferences with clinical evidence (Elwyn et al., 2012, Fisher et al., 2018). Both the patient and clinician are involved in all phases to share information and express treatment preferences, henceforth, they come to an agreement and make decisions together (Shay & Lafata, 2014). In long term conditions, this model is supported by the positive effects of involving patients in decisions, such as increased satisfaction with received health care, increased agreement with prescribed treatment, and quality of life (Durand et al., 2014; Kew, Malik, Aniruddhan, & Normansell, 2017). However, the implementation of SDM has been surprisingly hard to achieve in in routine mental health care and has not yet been implemented (Alguera-Lara, Dowsey, Ride, Kinder, & Castle, 2017; Slade, 2017).

There is increasing evidence that SDM positively impacts health outcomes (Huang, Plummer, Lam, & Cross, 2020; Joosten et al., 2008), but research on SDM in routine mental health care has a limited extent (Rodenburg-Vandenbussche et al., 2019, Slade, 2017). In clinical psychology and psychiatric care settings, SDM presents several research-related challenges to overcome an efficacious implementation and adequate assessment. To date, there is no gold standard measurement of SDM in research (Sepucha & Scholl, 2014). Even though there are an increasing number of tools to measure patient participation in treatment decisions (Gartner et al., 2018; Norful et al., 2020; Phillips, Street, & Haesler, 2016; Scholl et al., 2011), the measures contain many items, and observer-completed tools with coding scheme that may require training of scorers. Implementing the measurement of SDM in mental health clinical practice is also affected by comprehensive questionnaires that may not be practical for completion by patients in many clinical settings (Phillips et al., 2016), and by the reduced measurement quality of the questionnaires (Gartner et al., 2018, Phillips et al., 2016). The latter can affect the implementation process because studies from the field of clinical psychiatry are frequently described as busy mental health settings, where the most common perceived barrier is a lack of time (Huang et al., 2020; Pieterse, Stiggelbout, & Montori, 2019; Rodenburg-Vandenbussche et al., 2019), followed by the lack of adequate SDM tools (Kalsi, Ward, Lee, Fulford, & Handa, 2019). Another challenge regarding the implementation of SDM-measurement in clinical settings is that the most feasible assessment relies on self-reported patient experience measures. As such, extensive measures can be exhausting for patients, restricting their reliability and applicability. Appropriate and short measures are required to improve the implementation of SDM through valid and reliable assessment. However, only few studies have investigated tools to simplify the assessment of SDM in specialized mental health settings.

Thus, the choice of the most appropriate measure should be based on the instrument’s content and adequate characteristics, as this will help to avoid burdensome and time-consuming measures. The creator of CollaboRATE, a patient self-reported questionnaire developed in 2013, has proposed that CollaboRATE meets these requisites. The brevity and simplicity of the three-item version of the questionnaire has been emphasized in making it feasible to administer in different clinical settings. The questionnaire makes it easy to measure three core SDM dimensions: (1) the patient’s perception of how much effort was made to help them to understand their health issue; (2) how much the health professional listened to the patient concerning their health issue; and (3) how much effort was made to include what mattered most to most to the patient in the decision about what to do next.

The CollaboRATE focused on patients’ perceptions of being informed about and being involved in the decision‐making steps; these issues are briefly and clearly encapsulated in the questions. Moreover, it is sufficiently generic to be applicable to all clinical encounters and for all conditions (Elwyn et al., 2013), and it seems well suited for busy mental health settings.

Although the questionnaire was originally validated in 2013, the evidence of its validity has been increasingly documented among several populations. Barr et al. (2014) used data from the United States to report that the questionnaire had discriminant validity, concurrent validity, and sensitivity to change. Forcino et al. (2018) have pilot tested a Spanish translation of the questionnaire for use in the United States among patients attending an internal medicine clinic. Their study provided preliminary evidence of the acceptability for routine administration and documented that the questionnaire was easy to use (Forcino et al., 2018). Bravo, Contreras, Dois, and Villarroel (2018)) have conducted a validation study among women in maternity hospital units and found a one-factor solution for the modified version of the CollaboRate. Hurley et al. (2019) have compared the CollaboRATE with other extensive measures in a pediatric outpatient setting and found that parents of children aged 1-5 years preferred the CollaboRATE.

A Swedish validation study from 2016 (N = 121) was conducted among adults attending municipal services, supported housing services, and rehabilitation mental health programs (Rosenberg, Schon, Nyholm, Grim, & Svedberg, 2017). The study found that a modified version of the CollaboRATE displayed good face and content validity, adequate stability over time, and high internal consistency (Rosenberg et al., 2017). Another recent Swedish study conducted among patients with obstructive sleep apnea (N = 193) reported that the CollaboRATE showed good evidence of validity and reliability to measure SDM in addition to accessible time completion (less than 30 seconds) (Brostrom, Pakpour, Nilsen, Hedberg, & Ulander, 2019). A validation study conducted in Argentina among adults (N = 56) receiving treatment in family medicine and primary care centers (N = 30), and specialists (N = 26) found that the CollaboRATE presented adequate evidences of reliability and criterion validity (Ruiz Yanzi et al., 2019).

Although several studies have reported that the CollaboRATE is an appropriate tool for the assessment of SDM in routine practice, more research is required to further explore the appropriateness of the brief CollaboRATE. There is currently no evidence to support the suitability of the questionnaire for the specialized psychiatric care. Furthermore, the CollaboRATE has not yet been validated in Spain. Therefore, the aim of this study is to assess the psychometric properties of CollaboRATE as a measure of the SDM process among outpatients attending specialized in routine mental health care in Spain. The following main hypotheses were stated: The CollaboRATE would yield a good fit in a one-factor solution and a good internal consistency.

Method

Participants

This study cross-sectional study was conducted at the Canary Islands Community Mental Health Hospital. In the fourth quarter of 2019, 250 consecutive outpatients were invited to participate anonymously in the study, 191 persons accepted, and 23.60% refused to participate. Each day, a total of 15 randomly selected patients were informed about the study. In order to be selected, possible participants needed to match to following inclusion criteria: (a) be identified by the personnel at the outpatient service, as a person receiving treatment at the center; (b) be 18 years or older; (c) fluency in Spanish; and (d) consent to participate.

Instruments

The CollaboRATE measure is a three-item patient self-reported questionnaire developed by Elwyn et al. (2013) The items assess SDM. Responses to each item range from 0 (no effort was made) to a maximum of 9 (every effort was made) for a total of 27, with a higher score indicating a better patient-reported experience with SDM. CollaboRATE scores are calculated as the proportion of participants who report a score of nine on each of the three CollaboRATE questions. A calculation is also based on the percentage of patients who rated all three CollaboRATE questions. Similarly, it is possible to calculate the mean of the three CollaboRATE scores and the mean of the sum of the three items, with higher scores representing better self-reported experiences with SDM.

The Control Preferences Scale (CPS), developed by Degner, Sloan, and Venkatesh (1997)), measures the amount of control that patients want to assume in the process of making decisions about the treatment of their diseases was measured using the CPS (Degner et al., 1997). The card-sorting version of the scale was used in this study. It consists of five cards on a board; each card illustrates a different role in decision-making by means of a cartoon and short descriptive statement. In this study, the patients carried out two assessments: prior to the consultation, the examiner asked the patients to choose their preferred card; after being treated by the mental health professional, the examiner asked them to evaluate their experience using a modified version of the scale (De Las Cuevas & Peñate, 2016).

Procedure

Patients with an appointment with a psychiatrist at the hospital were invited to participate. A total of three psychiatrists were working at the center during the data collection period. Some of the patients self-reported that they were receiving treatment from a psychologist as well. The patients were identified by the personnel at the outpatient service, who informed them about the study and explained that those who were interested in participating could contact the researcher at the waiting room. Each day, a total of 15 randomly selected patients were informed about the study. All patients interested in participating in the study received full explanations of the study during their stay in the waiting room before the consultation. Those who consented to participate filled out the CollaboRATE and the CPS questionnaires before and after their consultations, together with a brief sociodemographic survey. The study was carried out in accordance with the code of ethics of the Declaration of Helsinki, and all procedures and consent forms were reviewed and approved by the Ethics Committee of the Canary Islands Health Service.

Statistical analyses

The data were analyzed using SPSS version 25 for Macintosh (IBM, 2017) and using R library psych (R Core Team, 2019, Revelle, 2018) with ULLRToolbox by Hernández and Betancort (2016). The participants were described with means, standard deviations, frequencies, and percentages. A Goodman and Kruskal's gamma coefficient was calculated to analyze the relationship between preferred and experienced roles. Mean item scores for each of the three CollaboRATE items and top scores were calculated. Calculations were also made for the proportion of desired and perceived participation in treatment decisions. Internal consistency was calculated using Cronbach’s α, Guttman's λ and ω. Factor analysis was performed using principal component and minimum rank analysis (Muñiz & Fonseca-Pedrero, 2019). The usual level of significance was set to p < .05, and 95% CIs were described where required to measure variability.

Results

Table 1 depicts the socio-demographic and clinical characteristics in addition to the preferences and experiences of participants, all according to the CPS. There were more female participants (63.40%), and the mean age of all participants was 44.90 ± 15.30 years. Only a minority of participants had higher education: 27.70% had a university degree. Patients’ diagnoses were available in 92% of the patients and included depressive disorder as the most prevalent (37.70%), followed by anxiety disorders (34.60%), schizophrenia (11.50%), and bipolar disorders (4.70%).

Table 1

Sociodemographic and clinical characteristics of the sample studied (N = 191).

	Global
Female, n (%)	121 (63.40)
Mean age (SD)	44.90 (15.30)
Education level, n (%)Primary studiesSecondary studiesUniversity degree	45 (23.60)93 (48.70)53 (27.70)
Time as psychiatric patient (months), mean (SD)	85.90 (109)
Diagnosis, n (%)SchizophreniaBipolar disorderDepressive disorderAnxiety disorderPersonality disorder	22 (11.50)9 (4.70)72 (37.70)66 (34.60)7 (3.70)
Preferred role according to CPS, n (%)I prefer to make the final treatment decisionI prefer to make the final treatment decision after seriously considering my doctor’s opinionI prefer that my doctor and I share responsibility for deciding which treatment is bestI prefer that my doctor makes the final treatment decision, but seriously considers my opinionI prefer to leave all treatment decisions to my doctor	3 (1.60)19 (9.90)87 (45.50)58 (30.40)24 (12.60)
Experienced role according to modified CPS, n (%)I made the final treatment decisionI made the final treatment decision after seriously considering my doctor’s opinionMy doctor and I shared responsibility for deciding which treatment was bestMy doctor made the final treatment decision, but seriously considered my opinionMy doctor made all treatment decisions	0 (0)8 (4.20)99 (51.80)55 (28.80)29 (15.20)

Almost half of the patients (n = 87, 45.50%) expressed their preference for the doctor and patient sharing responsibility for the decision about which treatment was best, while 24 (12.60%) preferred to leave all treatment decisions to their doctor. Only three patients (1.60%) preferred to make the final treatment decision themselves. Half of the sample expressed a collaborative attitude (n = 99, 51.80%) and self-reported that they shared responsibility with their doctor for deciding which treatment was the best. Next, 52 persons (28.80%) self-reported that their doctor made the final treatment decision after seriously considering their opinion. Overall concordance between preferred and experienced SDM was only 51.30% (Table 2). A total of 51 persons (26.70%) self-reported having a more passive role than preferred, and 42 (22%) self-reported having a role that was more active than they preferred. Goodman and Kruskal's gamma coefficient, which was calculated to analyze the relationship between preferred and experienced roles, exhibited a statistically significance concordance (gamma = .30, p = .003).

Table 2

Crosstabulation of preferences and experiences of shared decision-making according to the CPS.

	I made the final treatment decision	I made the final treatment decision after seriously considering my doctor’s opinion	My doctor and I shared responsibility for deciding which treatment was best	My doctor made the final treatment decision, but seriously considered my opinion	My doctor made all treatment decisions
I prefer to make the final treatment decision	0%	0%	33.30%	0%	66.60%
I prefer to make the final treatment decision after seriously considering my doctor’s opinion	0%	26.30%	42.10%	15.80%	15.80%
I prefer that my doctor and I share responsibility for deciding which treatment is best	0%	0%	67.80%	20.70%	11.50%
I prefer that my doctor makes the final treatment decision, but that they seriously consider my opinion	0%	5.20%	39.70%	44.80%	10.30%
I prefer to leave all treatment decisions to my doctor	0%	0%	33.30%	33.30%	33.30%

The CollaboRATE measure was welcomed by participating patients because it took less than 1 minute to complete and had no missing data. As Table 3 indicates, the mean item scores for each of the three CollaboRATE items ranged from 7.40 ± 2.10 to 7.60 ± 2.10. The top scores for each of the three items ranged from 47.10% to 51.80%, and 39.80% of psychiatric outpatients gave the best possible score on the three items. Neither socio-demographic (gender, age, educational level) nor clinical variables (diagnosis and time under psychiatric treatment) played a relevant role in patients’ perception of SDM according to CollaboRATE, since no significant differences were confirmed.

Table 3

CollaboRATE scores by item and total score.

CollaboRATE items		Mean ± SD	Top score*
Item 1	How much effort was made to help you understand your health issues?	7.50 ± 2.10	51.80%
Item 2	How much effort was made to listen to the things that matter most to you about your health issues?	7.60 ± 2.10	53.90%
Item 3	How much effort was made to include what matters most to you in choosing what to do next?	7.40 ± 2.10	47.10%
CollaboRATE Total Score		22.60 ± 6	39.80%

Note. SD: Standard deviation; *Top score, percentage of patients who gave the highest rating possible, which means 9 in every item and 27 in the total score.

Evidence of validity of internal structure of the CollaboRATE

The performed factor analysis, using principal component analysis, confirmed the unidimensionality of the CollaboRATE measure. The three items perfectly fit a single factor structure that explained 91.10% of variance and factor loadings between 0.94 and .96 (Kaiser-Meyer-Olkin Measure of Sampling Adequacy = .79; Bartlett's Test of Sphericity: Chi2 = 578,511, df = 3, p < .001; Principal Component Analysis Communalities: item 1 = .90, item 2 = .93, item 3 = .87). Table 4 shows the correlation matrix of the factor analysis.

Table 4

Correlation matrix of exploratory factor analysis of CollaboRATE items.

	Item 1	Item 2	Item 3
Item 1	-	.88***	.84***
Item 2	.88***	-	.87***
Item 3	.84***	.87***	-

p < .001

An exploratory factor analysis (EFA) using minimal rank solution was also performed. The solution obtained was compared with and EFA from Pearson correlation matrix following the Ordinary Least Squares (OLS) procedure with solution of minimal residues (Minimal Residual Solution). Both types of factor solutions had the same weights (CollaboRate01 = .92; CollaboRate02 = .96; CollaboRate03 = .90).

Reliability of the CollaboRATE

The CollaboRATE measure indicated a high internal consistency (α Cronbach = .95, Guttman's λ = .93, and ω = .95), which suggests that the scale measures only one unique concept. The three items registered high correlations with each other (.84 - .89), indicating that they are measuring the same construct. CollaboRATE items showed high correlations with the total scale score (.88 - .92). Corrected homogeneity or discrimination indexes obtained where really high (Item 1 = .89; Item 2 = .91; Item 3 = .88). No item deletion improved Cronbach's α.

Item response theory modelling for evaluating questionnaire item and scale properties was applied. A study of the instrument from a polytomous graduated response model (Graded Response Model) was carried out. A significant model evaluated was reached through an ANOVA model comparing the restricted model vs. unrestricted [AIC 1417.70; BIC 1515.20, Loglik -678.83 LRT 6.86, df 2, p < .032]. Fig. 1 display the test information function showing the goodness of the test when it comes to accurately measuring those of the latent trait for different levels of the attribute.

Fig. 1

CollaboRate test information function.

Table 5 depicts the CollaboRATE total score in each self-reported experience according to the CPS. The evidence of convergent validity, a subtype of internal structure validity, indicated that the more passive the experience according to the CPS, the lower the score obtained in the CollaboRATE measure.

Table 5

ANOVA results of CollaboRATE total score and experiences of control according modified Control Preferences Scale.

Experienced Control	n	%	Mean (SD)	ANOVA
I made the final treatment decision	8	4.2	25.90 ± 2.10	F = 111.22df 3, p < .001Eta = .80Eta Squared = .64
My doctor and I shared responsibility	99	51.8	25.20 ± 2.10
My doctor made the final treatment decision, but seriously considered my opinion	55	28.8	23.30 ± 4.60
My doctor made all treatment decisions	29	15.2	11.40 ± 5.60
Total	191	100	22.60 ± 6

Note. df: degrees of freedom

Discussion

This study is the first to examine the psychometric properties of the Spanish version of CollaboRATE as a SDM measure in specialized community mental health hospitals. This study yielded relevant results that supported the evidence of the validity of the CollaboRATE. The measure was welcomed in the community psychiatric care; in which a high percentage of outpatients agreed to complete the scale in a short time, and there were no missing data. The one-factor structure and the evidence of convergent validity demonstrated by strong positive correlations between the CollaboRATE and the CPS supported the evidence of validity of the CollaboRATE.

The high internal consistency reported in the present study is similar to those previously reported by Rosenberg et al. (2017); Bravo et al. (2018); Hurley et al. (2019), and Ruiz Yanzi et al. (2019). Our results confirmed the unidimensional construct of the questionnaire, and are consistent with the hypothesized factor reported in the original study (Elwyn et al., 2013). However, contrary to another study from Spain suggesting that SDM may differ depending on gender or medical condition (Calderon et al., 2018), in our sample, neither socio-demographic (gender, age, educational level) nor clinical variables (diagnosis and time under psychiatric treatment) played a relevant role in patients’ perception of SDM according to the CollaboRATE. On the other hand, the evidence of convergent validity of the instrument was proven by the significant differences evidenced in the CollaboRATE scores of the different perceptions of involvement, according to the modified CPS. Even though these results support the evidence of validity of the CollaboRATE, they also clearly indicate that SDM is not yet widely implemented across specialized community mental health hospitals in Spain.

Contrary to other studies (Moran-Sanchez, Gomez-Valles, Bernal-Lopez, & Perez-Carceles, 2019), more than half of the participants preferred an active role. However, less than half of the participants reported the best score possible on the CollaboRATE, and there is a limited concordance between the preferences and experiences of psychiatric outpatients in SDM. These findings are in line with previous studies suggesting that SDM has yet to be comprehensively implemented in Spain (Perestelo-Perez et al., 2011). This is evidenced by how the Spanish National Health Service still has not incorporated reforms in law that include SDM as a relevant component of mental health care services, or how there still is no standardized practice of interventions to reinforce decisions in routine mental health care (Perestelo-Perez et al., 2011).

Furthermore, SDM in mental healthcare and clinical psychological care delivery might require competent professionals with personal beliefs and preferences to negotiate an agreement with a patient who can make decisions and holds personal beliefs and preferences (Grim, Rosenberg, Svedberg, & Schon, 2016; Huang et al., 2020; Rodenburg-Vandenbussche et al., 2019; Simmons, Hetrick, & Jorm, 2010). Although psychologists and mental health professionals generally support SDM (Barr, Forcino, Mishra, Blitzer, & Elwyn, 2016; Chong, Aslani, & Chen, 2013; Hamann et al., 2009; Huang, Plummer, Lam, & Cross, 2020), they may also consider such approach to be an idealized view that is distant from standard psychiatric practice (Angell & Bolden, 2015), as well as condition-dependent or decision topic-dependent (Seale, Chaplin, Lelliott, & Quirk, 2006), and partially non-beneficial under certain circumstances (Hamann et al., 2009). On the other hand, patients in mental healthcare settings might demonstrate a variability in their preference for involvement (De las Cuevas & Peñate, 2014; De las Cuevas, Peñate, & de Rivera, 2014), and when the role preferences are not explored by therapists or clinical psychologists, it is difficult to practice based on concordance with the patient’s desire of involvement.

The practice of shared decision-making is related to multiple factors and several barriers. The review of the literature has identified time constraints and patients’ decisional incapacity as the main barriers that stand in the way of improving SDM implementation (Hofstede et al., 2013, Huang et al., 2020, Simmons et al., 2010). In mental healthcare, time is precious and scarce, and the lack of such time has been frequently reported as a significant barrier to SDM (Huang et al., 2020, Legare and Thompson-Leduc, 2014, Pieterse et al., 2019). However, the lack of time as a barrier is controversial because there are few studies supporting the claim that it takes too much time (Huang et al., 2020, Rodenburg-Vandenbussche et al., 2019). Anyway, mental health care systems should place a much higher value on and invest in innovations that create time and realize the possibility of time for patient care (Legare & Thompson-Leduc, 2014). Although there are studies that suggest that psychiatric patients’ decisional incapacity might be affected (Candia and Barba, 2011, Jeste et al., 2018), the majority of the patients are capable of making treatment decisions (Candia and Barba, 2011, Huang et al., 2020). However, the complexity of SDM is more challenging than the patients’ lack of competency to participate in decisions.

Half of the participants in this study reported that they experienced a lack of effort to help them to understand their health issue and to include in the decision-making process what mattered most for them. These statistics suggest that decision-making is not just about the patient’s ability or capacity to participate in decisions. The findings suggest that there is the need to reciprocate engagement to facilitate information about health issues, and that effort should be made to involve the patient in treatment planning. As the CollaboRATE is quick to fill out, it is a reliable and valid measure that can be used to routinely monitor, evaluate, and implement the model of SDM in specialized mental health settings.

This study invited a random sample of patients to participate, and used comparable questionnaires to collect information regarding patient-reported experiences about involvement in treatment decisions, which strengthened the internal structure validity of the CollaboRATE.

Although CollaboRATE has high applicability, the study has some limitations that need to be considered. First, the sample comprised psychiatric outpatients who attended at a single community mental health hospital. Second, this study had a high participation rate, and it is possible that a less motivated population could produce a different variability in the preference for their involvement and their experiences with the process of decision-making. In addition, this study had no clinical information of those who did not want to participate in the study, as well as there is a lack of information regarding possible comorbidities or socioeconomic data. These data could have been beneficial to carry on confounding analysis. Finally, due to the cross-sectional design of the study, it was not possible to evaluate the test-retest reliability of the measure, and such evidence of reliability testing should be the subject of future studies. Future research is also needed to investigate that psychometric property. In addition to test-retest reliability, the responsiveness of the questionnaire should be evaluated. It is also important to further assess the impact that comorbidity, received treatment and socio-demographic levels can have on the construct. Future studies could explore how the therapist characteristics affect the construct, and if the CollaboRate is sensitive to different therapists’ characteristics.

In conclusion, the present study suggests that the CollaboRATE is a psychometrically robust questionnaire in psychiatric clinical practice. The CollaboRATE measure has proven to be a clinically feasible tool for outpatients to estimate the level of SDM performed by their psychiatrists. It was also found that a significant number of patients are not involved in SDM. The use of this instrument can be crucial in developing knowledge of how best to implement SDM in clinical psychiatric routine care.

Authors contributions

The first and the last author contributed to the design of the study. All authors contributed toward drafting and critically revising the paper, gave final approval of the version to be published, and agreed to be accountable for all aspects of the work.

Conflict of Interest

The authors declare that there is no conflict of interest.