Measuring recovery related outcomes: a psychometric investigation of the recovery markers inventory.

The concept of recovery can be operationalized from either the point of view of the consumer or from the perspective of the provider of services. The Recovery Markers Inventory (RMI) was created to assess recovery-related factors (i.e., actions/events associated with consumer's recovery) from the provider's perspective. Evidence, which established the psychometric properties of the RMI, was obtained through the use of: (a) construct validity (i.e., confirmatory factor analysis and Rasch principal components analysis of residuals); (b) concurrent validity (i.e., the calculation of Pearson correlations between the RMI and other recovery-oriented instruments); and (c) reliability (i.e., Rasch Partial Credit models). Evidence presented in this article shows that the RMI scale is unidimensional, has an adequate level of correlation, and acceptable reliability. The current analysis provides evidence to support the RMI as a valid, reliable measure of recovery-related factors, which can complement consumer based instruments in the assessment of changes in recovery.

Spurred by increased interest of members of consumer advocacy groups and recent political influence, professionals in the field of community mental health have undertaken a systemic transformation in order to incorporate the ideas and concepts that facilitate consumers’ recovery (Olmos-Gallo et al. 2012). At the same time, mental health researchers and evaluators have focused on a change from the traditional outcomes (e.g., symptom frequency, recidivism, and hospitalization rates) to recovery-oriented outcomes (Davidson and Roe 2007; Farkas et al. 2005; Olmos-Gallo and DeRoche 2010). Typically, recovery outcomes include the assessment of change in consumers’ perspectives of recovery, commonly defined as changes in: hope, active growth, safety, symptom management, social support, and wellness among others (Lusczakoski et al. 2013; Onken et al. 2007). However, due to the deeply individualized nature of the recovery process (Anthony 2000), direct measurement of this process can be a challenge. In this article, we suggest that in addition to instruments that can be used to measure consumers’ assessment of their own recovery (Lusczakoski et al. 2013; O’Hare et al. 2003; Ridgway 2003; Trauer and Tobias 2004), providers can use measures related to the consumer’s recovery-related factors to assist in the assessment of change in recovery. This paper describes a measure to assess recovery related factors.

As implied by the term, recovery-related factors are actions and events that tend to be correlated with consumers’ recovery, even though the consumers may not necessarily associate them with their own personal journey. In that sense, recovery-related factors can be an indicator or marker of growth in recovery. For example, participation in mental health services may foster recovery, despite the fact that consumers can recover without the involvement/assistance of professional help (Anthony 2000). Similarly, a provider’s assessment of the consumer’s interest in employment or symptom management can be associated with the consumer’s recovery, even though recovery is dependent upon the consumer taking control and responsibility for his/her own life (Jacobson and Curtis 2000). Based upon this rationale, the provider’s assessment is defined as an evaluation of recovery-related factors and not as recovery, per se. In this article, we define recovery-related factors as including: employment, self-education/learning, participation in services, housing, symptom management, active growth, and substance use/abuse. Table 1 presents the operational definitions of these recovery-related factors.

Table 1

Operational definitions of recovery-related factors

Recovery-related factor	Operational definition
Employment	Actions toward looking for or maintaining employment, with markers of employment progressing from no interest in employment to searching for employment opportunities, supportive employment, extending up to full time independent employment
Education/learning	Actions undertaken to continuing education, with markers including the actions of looking up information on the internet, newspapers, and books, taking skills-oriented classes (e.g., cooking class), as well as vocational and formal education
Participation in services	Actions undertaken to self-direct a consumer’s recovery, ranging from not engaged in services to promote recovery, engagement, and the ideal level of directing own recovery (i.e., not associated with the frequency of services received)
Housing	Actions taken toward obtaining stable housing, with markers ranging from homelessness to residential housing, group homes, up to independent living
Symptom management	Consumers’ ability to cope with their symptoms within their daily lives ranging from No Impact (i.e., no impact on ability to function) to Very High (i.e., very high impact on the ability to interact with other people, or engage in work, etc.)
Active growth	Actions taken to seek and engage in activities within and outside the mental health center
Substance use/abuse	Consumers’ level of use across different substances and their stage of change (Prochaska et al. 1992) regarding their use

The Recovery Markers Inventory (RMI; developed by the authors) is a short survey completed by case managers/clinicians, intended to measure the recovery-related factors described in Table 1. Initial development of the RMI included a review of the literature and use of data collected from consumer focus groups. This process was supplemented by feedback provided by therapists, consumers, and members of the Mental Health Center of Denver’s (MHCD) Recovery Committee (see: Olmos-Gallo et al. 2012 for background about the MHCD Recovery Committee). RMI items have been used to generate a summed score, implying the measure is unidimensional, but no dimensionality analyses have been conducted to support that use.

The purpose of this study was to determine the unidimensionality, reliability, construct, and convergent validities of the RMI in a sample of adults with severe and persistent mental illness, who received services at a community mental health center at a major metropolitan city in the United States. We estimated reliability and unidimensionality using Rasch analysis (Bond and Fox 2001). We further tested unidimensionality conducting a confirmatory factor analysis in three steps, as suggested by Bollen (1989) and Tabachnick and Fidell (2013): (1) we split the original sample into two random halves, (2) fit a unidimensional model to one of the samples (calibration) and (3) cross-validated results with the second sample (validation). Finally, we tested convergent validity by calculating the correlation between the RMI and two measures of general functioning, and one intended to measure recovery from the consumer’s perspective.

Methods

Participants and Procedure1

A total of 1,513 consumers, who had RMIs completed by the case manager/clinician who regularly works with the consumer, in the MHCD Management Information System (MIS) during the month of July 2009, were included in the study. The MIS prompts clinicians/case managers every quarter to complete the instrument for every consumer in his/her caseload, and this reminder will not be removed from the to-do list until the RMI is completed. The RMI scores were matched through use of a unique ID number to demographic information and three other outcomes instruments: the Colorado Client Assessment Record (CCAR; Ellis et al. 1984); the Global Assessment of Functioning (GAF; Greenberg and Rosenheck 2005; Jones et al. 1995); and the Consumer Recovery Measure (CRM; Lusczakoski et al. 2013). These three instruments were completed within 30 days of the RMI. The CCAR is completed by the clinician/case manager who regularly works with the consumer; the GAF is completed by the psychiatrist who oversees the consumer’s treatment during the most recent visit, and the CRM is completed by the consumer him/herself. The participants’ demographic information is presented in Table 2.

Table 2

Participant demographics

Variable	Range	Mean (SD)
Age	18–95 years	44.07 (12.11)
Time in treatment at MHCD	0–240 months	68 months (72.5)

Instruments

Recovery Marker Inventory (RMI)

As described earlier, the RMI was designed to measure recovery-related factors defined as the provider’s assessment in the areas of employment, self-education/learning, participation in services, housing, symptom management, active growth, and substance use/abuse. With the exception of substance use/abuse, every area is measured by a single question with a different number of options. The different number of response options is intended to include all the significant measureable changes in the trait of interest. Thus, employment has six options, self-education/learning has 12, participation in services, 6; housing, 11; symptom management, 5; and finally, active growth orientation, has 6 options. It is worth noting that housing has 11 responses because they are also used for State and Federal reporting; however, for scoring purposes, the responses are collapsed into three general categories: (1) unstable/transitional housing (i.e., street, friends/motel homelessness); (2) stable housing (i.e., assisted living, congregate apartments, or single room occupancy); and (3) independent living. The assessment for substance use/abuse requires two steps. First, the provider assesses the consumer’s level of use for ten substances (Alcohol, Cannabis, Cocaine, Hallucinogens, Inhalants, Methamphetamines, Opiates, Over-the-counter, PCP, Sedatives/Hypnotics/Anxiolytics, and Stimulants). Second, if substance use is reported, the consumer’s stage of change is measured for the specific substance (Prochaska et al. 1992). Given that approximately one-half of all consumers report substance use/abuse (National Alliance on Mental Illness 2012), in the present study, the highest level of consumer use among all substances was applied as a moderator. The RMI was completed by the consumer’s regular case manager or clinician, through the on-line Medical Information System.

Colorado Client Assessment Record (CCAR)-Recovery and Global Assessment of Functioning (GAF)

The CCAR is the outcomes instrument used by the Division of Behavioral Health in the State of Colorado to assess mental health (Ellis et al. 1984), and is administered on a scheduled annual basis to all MHCD consumers. A revised version of this instrument includes items which measure: social support, hope, activity involvement, empowerment, and interpersonal relationships. A 9 point rating response scale is used, and this instrument is termed, the CCAR recovery scale (Menefee, personal communication, April 18, 2008). For the current sample, the CCAR recovery scale showed acceptable internal consistency (α = .85).

The GAF scale (Greenberg and Rosenheck 2005; Jones et al. 1995) is used by psychiatrists at MHCD to rate consumers’ functioning on a regular basis (i.e., at least once a year). The GAF is administered as part of the assessment submitted to the Division of Behavioral Health. The scores range from 0 (i.e., lowest functioning) to 100 (i.e., highest functioning). The GAF scores are used to assess overall functioning for individuals who struggle with their mental health, and they are used to assess the concurrent validity of other recovery scales (Fisher et al. 2009). The Office of Behavioral Health at the State level provides training as part of their CCAR package, but no estimates of the reliability of the GAF are yet available. However, strong, statistically significant correlations have been observed against the CCAR’s symptom severity, and overall level of functioning scales (Mahalik personal communication, February 2014). Although no intraclass reliability has been calculated, psychiatrists receive regular refreshers on scoring.

Consumer Recovery Measure (CRM)

The Consumer Recovery Measure (CRM: Lusczakoski et al. 2013) is used to evaluate consumers’ perceptions of their own recovery, and it is completed by consumers every quarter-year at MHCD. The survey consists of 17, 4 point rating response items, from Strongly Agree to Strongly Disagree. It is used to rate consumers’ self-perceptions of hope, safety, symptom management, social network, and active growth. The instrument was developed with use of a Rasch rating scale model (Bond and Fox 2001) and has a person separation reliability of 0.83 and an item separation reliability of 0.96 (Lusczakoski et al. 2013).

Data Analysis

To estimate the psychometric properties of the RMI, three analytical techniques were applied. These were: (a) reliability estimation and construct validity using Rasch analysis; (b) construct validity using confirmatory factor analysis (CFA). We randomly split the original sample into two halves, then fit a unidimensional model to one of the samples (calibration) and cross-validated the results with the second sample (validation); and (c) concurrent validity through Pearson correlations between the RMI and the CCAR Recovery Scale (Menefee 2008, personal communication), the GAF (Greenberg and Rosenheck 2005), and the CRM (Luzczakoski et. al. 2012).

First, a Rasch Partial Credit Model was estimated with the use of Winsteps 3.64 (Linacre, 2007) to determine the person and item reliability and dimensionality, as well as in fit and outfit estimates (Bond and Fox 2001).

Next, the CFA models were estimated with use of LISREL 8.80 (Jöreskog and Sörbom 2006), based on the procedures suggested by Bollen (1989) and Tabachnick and Fidell (2013). CFA is a confirmatory technique used to test a theory about measure structure (Tabachnick and Fidell 2013); in the present case, a single factor defining recovery-related factors. Participants’ were randomly divided into two datasets: (a) a calibration sample (n = 737) to estimate the structure components and (b) a cross-validation sample (n = 776) to verify them (Bollen 1989; Tabachnick and Fidell 2013). To account for the ordinal nature of the items, both CFA models were estimated using polychoric correlation matrices with robust maximum likelihood estimation procedures. Multivariate normality was evaluated and found to hold across all items. Model fit was assessed using Hu and Bentler’s (1999) guidelines: non-normed fit index ≥.95 (NNFI; Bentler and Bonett 1980); comparative fit index ≥.96 (CFI; Bentler 1990); and root mean square error of approximation (RMSEA; Steiger 1989) ≤.05 indicating good fit or ≤.08 for reasonable fit (MacCallum et al. 1996). Finally, Pearson-product moment correlations were computed between the RMI and the CCAR recovery scale, GAF score, and the MHCD CRM with use of SPSS 13 (SPSS Inc 2005).

Results

Rasch Analysis

The RMI showed acceptable reliability estimates, with a person separation reliability of 0.74 and an item separation reliability of 1.00. As displayed in Table 3, all six items in the RMI produced in fit and outfit values within an acceptable range (0.5–1.5). In terms of difficulty to endorse, self-education and employment were the two most difficult items, followed by active growth, participation, symptoms, and housing. As evidence of the unidimensionality, the strength of the first contrast following a principal components analysis of residuals was less than the suggested cutoff value of 2.0 (Bond and Fox 2001; Linacre 2007). In terms of model misfit, the majority of consumers’ Rasch model misfit was due to unexpectedly low scores in the housing item. That is, these consumers improved in every other area, but were still homeless or living in transitional housing.

Table 3

Infit, outfit, and difficulty estimates for the RMI items

Items	Infit	Outfit	Item difficulty
Self-education	.89	1.05	1.13
Employment	1.04	.98	.90
Participation	1.01	1.02	−.01
Active growth orientation	.82	.83	−.03
Symptom management	.94	.95	−.37
Housing	1.26	1.48	−1.62
Average	.99	1.05	–

Construct Validity

The specification of the calibration model included a single latent factor (i.e., recovery-related factor). Regarding substance use, the highest level among all ten substances served as a moderator variable (i.e., exogenous variable). The CFA model structure is presented in Fig. 1. The consumer’s highest level of substance use was negatively correlated to the recovery-related factor (r = −25, p < .01), which suggests that, as a consumer’s substance use increases, the recovery-related factors decrease. Given that the fit statistics were acceptable in the calibration model (NNFI = .96, CFI = .97, RMSEA = .06, CI = .04, .08), the model for the validation sample was specified according to the factor loadings and variance–covariance matrices estimated in the calibration model. The fit of the validation model was also acceptable (NNFI = 1.01, CFI = 1.00, RMSEA < .01, CI < .01, .01). Furthermore, a comparison between the validation and the calibration models showed no significant deviations (S–B χ2 = 0.0 (df = 28), p > .05). The findings from the construct validity analysis supported by Rasch analysis provided evidence that the six items in the RMI can be combined to create a total recovery-related latent factor score, which is moderated by a consumer’s highest level of substance use.

Fig. 1

CFA model with standardized factor load. Boxes represent observed variables, the large circle represents the latent variable and the small, dark circles represent measurement errors associated with observed variables

Concurrent Validity Evidence

The RMI exhibited significant correlations with other recovery measures. The strongest relationship was with the CCAR recovery scale (r = −.46, p < .01, 95 % CI −.39, −.52). The RMI showed statistically significant correlations with the GAF (r = .23, p < .01, 95 % CI .15, .32) and the CRM (r = .19, p < .01, 95 % CI .12, .25). It is worth noting that the negative correlation with the CCAR is due to the fact that the CCAR is used to measure severity, not recovery (Fig. 2).

Fig. 2

Item/person map of the Recovery Markers Inventory. Note The distribution of consumers’ ability scores are shown on the left side of the line; difficulty scores are depicted on the right side of the ruler

Discussion

The purpose of the study was to investigate the psychometric properties of the RMI, an instrument used to measure the factors associated with recovery from mental illness. As explained in the introduction, the RMI is not intended to measure recovery, but actions and events associated with growth that can be associated with it.

Evidence was provided by this study to support the RMI as a valid and reliable measure of recovery-related factors. The results from this study demonstrated that, with a community-based sample of consumers, the RMI showed (acceptable internal reliability estimates, unidimensionality as a measure of recovery-related factors, and evidence for concurrent validity in the measurement of consumer recovery and general overall functioning. Since the RMI has been used only with consumers receiving services at community-based mental health centers, it is suggested that similar settings can effectively adopt it. It is also worth notice that several of the indicators included under the RMI have been used on a regular basis to report outcomes to the Federal and State governments. Thus, the RMI can play a dual role by: (a) creating an accountability system for external stakeholders and funders; and (b) helping clinical staff, consumers, and their family members to observe improvements over time.

The RMI represents one of several instruments in the multi-pronged measurement approach undertaken by the Mental Health Center of Denver (MHCD) to understand recovery from mental illness. As described elsewhere (Olmos-Gallo et al. 2012), we have to date developed several instruments to explore recovery from multiple perspectives, in a way that can be easily integrated into clinical practice. The combination of multiple instruments has allowed MHCD staff to develop an outcomes-centered approach to recovery, which has already helped to demonstrate significant improvements. For example, since MHCD started to track recovery, with use of the RMI and CRM (Lusczakoski et al. 2013), staff of the two succeed in Employment program has reported a significant increase (38 %) in employment status (Olmos-Gallo et al.), and over 25 % obtained employment within 8 months of enrollment in the program. Approximately 55 % of the consumers in the same program increased their interest in education/learning activities (Outcomes Quarterly, Fall 2010).

The MHCD multi-measure approach has been used to develop a holistic approach toward recovery that takes into account the fact that recovery happens over time, and that there is always the possibility of relapse (Olmos-Gallo and DeRoche 2010). We have developed models with use of hierarchical linear modeling (Raudenbush and Bryk 2002), which can accommodate the hierarchical nature of the longitudinal data collected,2 in order to explore the nature of change over time and the potential impact of program characteristics (e.g., modality, intensity) or background variables (e.g., gender, ethnicity, diagnostic). More recently, we have added the use of statistical control charts (Montgomery 2008) that allow multiple stakeholders (e.g., managers, clinicians, consumers) to monitor and provide feedback on consumers’ progress as they move forward in their recovery. Outcomes information is combined with other relevant information (e.g., diagnosis, demographic) in a dashboard that presents it in a user-friendly manner. The combination of valid and reliable recovery-outcome measures, development of predictive models, and statistical control charts has helped MHCD staff to monitor clinical data with an emphasis on quality improvement (McLean et al. 2010).

More importantly, being able to measure recovery from different perspectives allows the staff to ask questions that can help them in their practice. For example: Are medications just relieving symptoms, or helping people improve? Are there some interventions that are better suited to consumers with specific characteristics? What kinds of services help the most when it comes to improving the lives of those with a severe and persistent mental illness? What is the impact of hope in recovery? The answer to these and similar questions will not be possible in the absence of valid and reliable instruments to measure recovery-based outcomes. In this study, we have shown that the RMI holds promise as a reliable and valid measure of factors associated with recovery from mental illness that can be used to answer such questions.